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INTRODUCTION BY HEATHER T. ROWAN-KENYON 


There are large gaps in educational access and attainment between the rich and poor in the 
United States. In 2016, there was a 25 percentage-point gap in college enrollment rates for high 
school graduates in the top and bottom family income quartiles. There were also gaps in 
attainment rates, with only an estimated 11 percent of dependent students from the lowest family 
income quartile attaining a bachelor’s degree by age 24, compared with 58 percent of students 
from the top family income quartile (Cahalan, Perna, Yamashita, Wright, Santillan, Pell Institute 
and PennAHEAD, 2018). 


Reflections on Connecting Research and Practice in College Access and Success 
Programs is the result of a collaboration between the Association for the Study of Higher 
Education (ASHE) and the Pell Institute for the Study of Opportunity in Higher Education (Pell 
Institute). ASHE received a grant from the William T. Grant Foundation to explore 
collaborations with the Pell Institute and four additional partner organizations, including the 
Council of Independent Colleges, NAFSA: Association of International Educators, the National 
Association of Student Financial Aid Administrators (NASFAA), and the Western Interstate 
Commission on Higher Education (WICHE). The goal of the ASHE/Pell collaboration was to 
address how researchers and practitioners can best work together to address their mutual goal of 
reducing these gaps and fostering increased opportunity for college access and success for all 
students and especially for low-income, first-generation students and students with disabilities 
that federal programs such as TRIO and GEAR UP as well as many other programs at the state 
and local levels are intended to serve. 


The overall grant was based on the premise that too often researchers conduct studies that have 
little relevance to policymakers and practitioners, and policymakers and practitioners are too 
often unaware of the relevant research that does exist. The aim of this initiative was to improve 
research-and-practice connections by encouraging collaborations between members of our higher 
education research association (ASHE) and members of each of five organizations that seek to 
help higher education policymakers and/or practitioners identify, understand, and apply research- 
based evidence. Intermediary organizations can play an important role in connecting research 
and practice, but frequently these intermediary organizations have limited capacity to effectively 
play this role. 


The goal of our specific collaboration between ASHE and the Pell Institute is focused on 
exploring links between research and practice in TRIO and GEAR UP programs to increase 
college access and attainment. In particular, we focused on how researchers and college access 
and success practitioners can best work together to inform one another and promote better 
outcomes for students. While our examples are primarily from the federal TRIO/GEAR UP 
programs, it is our hope that this compilation will inform the work of researchers and 
practitioners who are working generally in the field of college access and success, especially 
those with a focus on low-income, first-generation students and students with disabilities. 

Some of the primary questions that emerged from our working group include: 
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e How do we know what works, when, in what context, and for which students? 

e How can TRIO and GEAR UP practitioners make the best use of the extensive tracking 
data required for performance reporting in a manner that gets beyond compliance? The data 
has detailed individual student records of the outcomes of the students that are served. We 
ask the question of how this data can be used to institutionalize using data for program 
improvement. 

e How do we bring together external and internal evaluators to promote internal program 
improvement and external program evaluation? 

¢ How do we build relationships and trust between researchers and practitioners? 

e What is the map of practitioner knowledge related to research and evaluation? 


This essay collectionis comprised of contributions by researchers and_ practitioners 
based on some of the above themes. In the first chapter, Historical Reflections and Thoughts 
Moving Forward on Connecting Research and Practice, Laura Perna from the University of 
Pennsylvania focuses on inequality in connecting research and practice. Paul Beasley from the 
University of South Carolina shares some historical perspectives of TRIO programs and 
connecting research and practice in college access. Then, Margaret Cahalan from the Pell 
Institute for the Study of Opportunity in Higher Education shares 16 lessons for researchers and 
practitioners from the Mathematica Policy Research’s National Evaluation of Upward Bound 
Study. 


Chapter two, How to Make Collaboration Work, is composed of seven essays focused on 
researcher and practitioner collaborations. Judy Marquez Kiyama from the University of Denver 
and Kristan Venegas from the University of Southern California each talk about building trust 
between researchers and practitioners. Ezekiel Kimball, Tyson Rose, Yedalis Ruiz, and Ryan 
Wells illustrate one of these partnerships with researchers and practitioners from the University 
of Massachusetts Amherst and share lessons learned from their work. Angela Bell, Robert 
Anderson, Georgia Hughes-Webb, and Adam Greenexplore why relationships between 
researchers and staff of college access and success programs can be difficult but argue that the 
benefits for all are significant, using an example of the West Virginia GEAR UP program 
evaluation. Christopher Mullin from Strong Start to the Finish examines how to structure 
programs to acknowledge both program improvement and excellence, with a specific focus on 
leadership and management. Oscar Felix from Colorado State University and past COE Board 
Chair talks about formulating communities of practice to better link research and practice in the 
TRIO community. Mika Yamashita from the Pell Institute for the Study of Opportunity in 
Higher Education discusses the origin of the competitive preference priorities that appeared in 
Student Support Services and Talent Search grant applications in recent years and urges TRIO 
programs to articulate their own specific logics of evidence use that reflect how TRIO 
professionals work in practice. 


In our final chapter, The Relationship between Research and Practice: The Recent 
Landscape, Margaret Cahalan explores the implications of the What Works Clearinghouse and 
the Competitive Preference Priorities (CPPs) for TRIO programs and services. Mika 
Yamashita then shares our attempt to map the terrain of practitioner knowledge related to 
research and evaluation. Members of our working group conducted an on-line survey of TRIO 
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project directors to map the landscape of practitioner use of data and evaluation, issues of 
collaboration between practitioners and academics and contractors--- and to some extent tried to 
map the substantive knowledge base of practitioners. We had 80 TRIO program practitioners 
respond to our 80-question survey, and Mika shares these findings in her piece. Finally, Heather 
Rowan-Kenyon, Mika Yamashita, and Margaret Cahalan provide a wrap-up of the work of the 
collaboration. 
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CHAPTER I 


HISTORICAL 
REFLECTIONS AND 
THOUGHTS MOVING 
FORWARD ON 
CONNETCING RESEARCH 
AND PRACTICE 


IMPROVING CONNECTIONS BETWEEN RESEARCH 


PAN D PRACTICE BY LAURA W. PERNA * 


Introduction 
Improving college access and success for students from 
Abstract low-income families and students who are the first in 
their families to attend college requires a multi-faceted, 
This essay first reflects on the comprehensive approach, and commitment from 
differences between researchers multiple players (Perna & Jones, 2013). Among the 
and practitioners and _ then important players are the federally-sponsored TRIO 
offers suggestions for ways to programs. Research demonstrates the positive effects of 
improve connections between TRIO programs on students’ college-related outcomes 
TRIO. program research and (Maynard et al., 2014). Methodologically rigorous 
practice. It concludes by research studies conducted by Westat and Mathematica 
offering additional Policy Research show that: Student Support Services 
recommendations for ways that promotes persistence in college, college credit accrual, 


and college grades; Talent Search increases applications 
for financial aid and postsecondary enrollment; and 
Upward Bound Math-Science has positive effects on 
enrollment in selective four-year institutions and 
completion of a bachelor’s degree in a math or science 

discipline (The Pell Institute, 2009). In a meta-analysis 
La of research that used experimental or quasi- 


- -=- experimental research designs, Maynard et al. (2014) 


found that, on average, the studied TRIO and other 
college access programs increased college enrollment by 12 percentage points.” Other research 
demonstrates the cost-effectiveness of Talent Search, especially relative to other dropout 
prevention programs, in promoting high school completion (Levin et al., 2012). 


TRIO practitioners and 
academic researchers can 
collaborate to accomplish 
shared goals for first-generation 
and low-income students. 


' This essay draws from testimony I presented to the Subcommittee on Higher Education and Workforce Training, 
Committee on Education and the Workforce, United States House of Representatives, Washington, DC on April 30, 
2015 and from a meeting entitled, Improving the connection between research and state policy for increased 
attainment, I co-convened with the Lumina Foundation in Washington, DC on February 20, 2015. During this 
meeting, a small group of academic researchers and leaders of “intermediary organizations” discussed strategies for 
improving connections between academic research and state policymaking. Intermediary organizations can play an 
important role in connecting academic research and policy/practice, as these organizations tend to conduct and/or 
sponsor their own research projects and translate research findings to policymakers and practitioners. The Pell 
Institute may be considered an intermediary organization, as it seeks to help public policymakers and TRIO 
practitioners identify, understand, and apply research-based evidence to improve TRIO program practice. 
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Despite this research evidence, there is much that we do not know about “what works” among 
TRIO programs. To maximize the benefits of college access and success programs to student 
outcomes, policymakers and practitioners need to know which components and services work, 
for which groups of students, in which contexts (Perna, 2002). In their comprehensive meta- 
analysis of research on the effects of college access programs on college readiness and/or college 
enrollment, Maynard and colleagues (2014) identified only 34 studies that were published 
between 1990 and 2013 that used experimental or quasi-experimental research designs. Of the 
34 studies, 18 provided sufficient information to conduct a cross-study review of effects of 
targeted interventions on college readiness and/or enrollment (Maynard et al., 2014). This is a 
remarkably low number, given the large number of TRIO and other college access programs that 
are operating across the nation. Even fewer studies have attempted to identify the effects on 
college-related outcomes of particular program components and services (Maynard et al., 2014). 


Why are there so few studies that meet these criteria and address the knowledge needs of 
policymakers and practitioners? Lack of interest would not seem to be the problem: TRIO 
program leaders and administrators want to use practices that are known to produce meaningful 
improvements in the college-related outcomes of the students participating in their programs. 
Academic researchers want to produce high-quality research that effectively demonstrates the 
effects of particular practices on student outcomes. 


Nonetheless, despite overlapping goals and interests of TRIO practitioners and academic 
researchers, differences in incentives, approaches to objectives, and time horizons between the 
two groups may limit the extent to which their shared goal is achieved. This essay first reflects 
on these differences and then offers suggestions for ways to improve connections between TRIO 
program research and practice. 


Differences between Academic Researchers and TRIO 
Program Practitioners 


Academic researchers and TRIO program practitioners differ in many ways. One difference 
pertains to orientations and motivations. Academic researchers are trained to identify 
implications for theory, identify the many contextual forces and limitations that influence results, 
and employ research designs that require extensive time to implement and complete. Academic 
researchers are incentivized to publish articles in academic journals, a process that may delay the 
dissemination of research-based findings until months or years after the research has been 
conducted. 


° This meta-analysis includes a controversial study that found that Upward Bound has “no detectable effect” on 
college enrollment (Seftor, Mamun, & Schirm, 2009). Reevaluations of data from this study show that, when design 
flaws of the Seftor et al. study are taken into account, Upward Bound has positive effects on college enrollment, 
college completion, and applications for financial aid (Cahalan & Goodwin, 2014; Harris, Nathan, & Marksteiner, 
2014). Including the reevaluation of Upward Bound rather than the original Upward Bound study in the Maynard et 
al. meta-analysis increases the pooled effect of the studied programs on college enrollment from 11.9 percentage 
points to 12.2 percentage points. 


TRIO program leaders and practitioners are motivated to take immediate action, implementing 
the practices that they believe to be beneficial to the particular students that they serve. Whereas 
academic researchers strive for objectivity, TRIO program leaders and practitioners may prefer 
practices that are relatively easy to implement, familiar to program staff, and perceived to be 
effective by students and other stakeholders. 


TRIO program practitioners and academic researchers also differ in terms of their access to and 
perspectives on relevant data. TRIO practitioners control the collection and management of 
programmatic data. With a primary focus on delivering services, TRIO programs typically have 
relatively few human or other resources for in-house data collection and analysis. Academic 
researchers have the methodological training and resources to analyze the data but may not have 
the connections with TRIO programs that are required to gain access to needed data or 
understand the logistical challenges of collecting different data. 


Creating the potential for uneven power dynamics, TRIO program administrators and academic 
researchers also tend to differ in terms of institutional resources and other sources of support. 
TRIO program administrators may have fewer institutional resources on which to draw than 
academic researchers. Academic researchers also tend to be more highly paid than the TRIO 
practitioners who are delivering program services. 


Improving Connections between TRIO Research and Practice 


Creating meaningful improvements in the connections between TRIO research and practice 
requires recognition of these and other differences in incentives, motivations, time horizons, and 
resources. This section suggests strategies for recognizing these differences and, consequently, 
helping to ensure that TRIO practitioners and academic researchers achieve the shared goal of 
identifying the most effective use of available resources for improving college access and 
success for low-income and first-generation students. 


Implicit in these suggestions is the assumption that both academic researchers and TRIO 
practitioners have assets and strengths that should be capitalized upon. While academic 
researchers possess expertise in the procedures for conducting research, TRIO program 
practitioners are experts in what the programs seek to accomplish, for whom, under what 
constraints. 


TRIO program practitioners who seek to improve connections between research and practice 
should: 


1. Request resources in funding proposals to support and advance data collection and research; 
and 


2. Involve researchers early in the process of conceptualizing data and research efforts. 


TRIO program administrators are incentivized to collect data that satisfies government 
accountability demands. But, academic researchers often find these data alone to be insufficient 
for conducting methodologically rigorous research studies. By involving researchers early in the 
process of conceptualizing data and research, and by requesting resources to advance data 


10 


collection and research in funding proposals, TRIO programs will help to productively advance 
connections between research and practice. 


Higher education researchers who seek to promote better connections between research and 
practice should: 


1. Conduct research that addresses the knowledge-needs of TRIO practitioners; 
2. Empower practitioners to partner in the conduct of research; and 


3. Make research results available and accessible to TRIO practitioners and policymakers. 


For academic research to have a meaningful impact on TRIO practice, it must be relevant to the 
knowledge needs of TRIO practitioners. Practitioners often have questions about the practices 
that produce desired outcomes, the design and implementation of effective practices, and 
variations in effects of practices across groups of students and contexts. Practitioners also want 
to know what they think that they don’t already know. Academic researchers and TRIO 
practitioners need to engage with each other to achieve shared understanding of knowledge 
needs and research goals. 


Academic researchers should also work to empower practitioners in the research process. Rather 
than treating TRIO program practitioners as subjects to be studied, academic researchers should 
consider opportunities for action research and other research approaches that involve TRIO 
practitioners as partners in the conduct of the research. 


Academic researchers need to not only learn the knowledge needs of TRIO practitioners, but also 
make the results of their research accessible. TRIO program leaders and administrators — and the 
policymakers to whom they are accountable — often do not have access to scholarly journals, or 
the time to read lengthy manuscripts. Academic researchers should consider disseminating 
results in outlets that are typically read by TRIO practitioners and producing short, easy-to-digest 
summaries with links to additional information. 


Role of Foundations and Scholarly Associations in Promoting 
Connections between TRIO Research and Practice 


Foundations and academic research associations can also play a role in promoting connections 
between academic researchers and TRIO practitioners. Typically only academic researchers 
attend and participate in conversations at academic/scholarly research conferences. Recognizing 
this structural reality, the William T. Grant Foundation provided a small grant to ASHE to 
encourage collaborations among ASHE members (researchers) and members of external 
organizations that are oriented toward serving policymakers and practitioners. As part of this 
project, Heather Rowan-Kenyon (Associate Professor at Boston College and ASHE member) 
and Margaret Cahalan (Director of the Pell Institute) have been actively engaged in promoting 
discussions among ASHE researchers and TRIO practitioner. As then president of ASHE and PI 
on the William T. Grant Foundation grant, I am excited about the ways in which this 
collaboration (along with collaborations between ASHE members and representatives of four 
other intermediary organizations) are helping academic researchers and practitioners develop 
shared research priorities and other shared goals and outcomes. 
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More research-based knowledge is needed about best practices for promoting college-related 
outcomes for low-income and first-generation students along the college-going pipeline, from 
middle-school into post-graduate study, and for both traditional-age students and adults who 
aspire to attend and complete college. By recognizing and capitalizing on differences in 
perspectives and resources, and by intentionally acting to build bridges between research and 
practice, researchers and practitioners are more likely to accomplish the shared goal of ensuring 
that available resources are effectively used to improve college-related outcomes for low-income 
and first-generation students. 
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CONNECTING PRACTITIONERS AND RESEARCHERS 


TO STRENGTHEN COLLEGE ACCESS AND SUCCESS 
mele AVANABS) BY PAUL BEASLEY 


Connecting research and practice in college access 
and success programs first requires connecting 
researchers and practitioners. The usefulness of 
high-quality research to the development of solutions 
to problems as complex as those that create the need 
for college access programs is undeniable. But so is 
the involvement of practitioners who not only 
understand the objective dimensions of effective 
interventions but also the importance’ of 
acknowledging and preserving the dignity of clients. 
Research that places concern for its structure above 
concern for the dignity of participants is inherently 
flawed regardless of its approach. Working with 
practitioners who have appropriate sensitivity and 
respect for clients as well as knowledge of 
experimental design can help to ensure the 
development of high-quality studies that are useful to 
practitioners and fair and ethical to the programs and 
individuals they are supposed to help. 


A connection between quality research and practice 
in college access and success programs would align 
the work of college access programs with that of 
other professions in which clients depend on 
knowledgeable practitioners to guide them in making 
critical decisions about their lives. Just as it is 
reasonable to expect doctors, social workers, and 
psychologists to have appropriate expertise guided 
by up-to-date research, it is also reasonable to expect 
college access professionals to use these components 
of evidence-based practice in their work with 
children and young adults. Decisions about 
attending college can be as consequential as 
decisions about health, especially for disadvantaged 
students for whom a college education is not an 
option but a necessity if they are to achieve an 
adequate standard of living in the twenty-first 
century. Wages for poorly educated and less-skilled 
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Abstract 


TRIO. programs have had a 
complex relationship with research 
which illustrates some of the 
problems that must be addressed to 
connect research and practice in 
college access programs. This 
paper explores that relationship 
from the vantage point of over 40 
years of experience in TRIO 
administration. It reviews the 
impact of federal regulations on 
program development and_ the 
consequences of several national 
studies that used both experimental 
and quasi-experimental designs. 
This paper concludes with ideas 
for strengthening the link between 
research and practice in college 
access that emphasizes 
practitioners as full partners in the 
process. It calls on government, 
professional associations, and 
higher education to facilitate these 
partnerships to encourage high 
quality research studies that focus 
on identifying effective strategies 
and that show evident respect for 
the —_ programs and their 
participants. 


workers are already below the poverty level, and employment projections favor only those with 
high-level skills. Since most poor and working class students have inadequate academic 
preparation and high financial need, it is unlikely that they can succeed in college without proper 
guidance. In fact, uninformed and unsuccessful attempts to earn college degrees can create the 
additional hardship of wasted financial aid eligibility and loan indebtedness that can last a 
lifetime. Providing help with these challenges is the important work of college access programs, 
and this work should be guided by evidence-based practice with expectations for high-level 
expertise supported by rigorous research and implemented with high regard for students. 


The absence of a strong connection between TRIO college access programs and research is not 
surprising, even though most of these programs are based in higher education where a substantial 
amount of research takes place. College access did not develop as education theory but as 
political advocacy during the Civil Rights Movement. As political policy, funding for programs 
realistically depended on effective work within the halls of Congress and state legislatures that 
did not necessarily involve research. The influence of successful political advocacy has had a 
lasting impact on the programs and the people who work in them. TRIO practitioners are 
organized within a network of organizations led by the Council for Opportunity in Education 
(COE) and committed to strengthening program and staff capabilities, building collaborations 
with public and private interests to improve and expand college access, and, most noticeably, 
encouraging congressional support for college access and success. 


Advocacy during the Civil Rights Movement led to the Higher Education Act of 1965, which 
also had a tremendous impact on the early development of college access in both meaning and 
operation. This legislation contained several grant programs, including the TRIO Programs, 
which made the federal government the largest provider of support for college access services. 
The size and scope of these programs gave significant influence to the rules and regulations 
developed by the Department of Education for grant administration. Compliance became an 
important concern to those interested in continuous funding since the loss of funding eligibility 
most often resulted from noncompliance. 


Concern about compliance with federal rules had a particular impact on the use of research. 
Within the grant application process, applicants had to certify that grant funds would not be used 
for research that involved human subjects. This restriction was often explained by federal 
program officers as a strict prohibition on all research. Eager to avoid violation of rules that 
determined eligibility for continued funding, program mangers excluded all activities and 
expenditures associated with research activities. 


Coupled with this perceived restriction on research were regulations that limited participation in 
grant activities to the small number of program participants who were often only a fraction of a 
college’s or school’s student population. This resulted in most programs operating in isolation 
from the mainstream of student activity, and being nearly invisible to the majority of faculty and 
staff who did not interact with them. Although later revisions in the regulations allowed for 
some integration of programs services with the general administration of campus activities, this 
early isolation of programs on campus, and their lack of interaction with general student 
concerns, resulted in limited opportunities for TRIO Programs to develop relationships that 
might lead to collaborations with practitioners and researchers. 
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Federal grant administration and an emphasis on compliance also meant accepting the particular 
notion of college access spelled out in funding criteria, which for TRIO Programs have not 
changed substantially in the past 50 years. This has resulted not only in stagnant program 
development but also in a uniform approach to college access programming across the broad 
spectrum of higher education. For example, the grant application for the TRIO Student Support 
Services Program requires community colleges, regional campuses, research universities, and all 
other eligible institutions to work from the same parameters for program development with only 
few exceptions for institutional type. This uniform approach is based in research from the 60s 
and 70s that most often conceived of college access programs as remedial services. Beyond 
reauthorization of the Higher Education Act, few opportunities exist to make changes in this 
process. 


During the 1990s and 2000s, the U.S. Department of Education used random assignment and 
quasi-experimental studies to evaluate the effectiveness of several TRIO programs. These 
studies were conducted under large contracts with national firms. Mathematica Policy Research 
Inc. (2009) conducted a longitudinal random assignment study of Upward Bound. It also 
conducted a quasi-experimental study of Talent Search (2006) that was limited to three states, 
Texas, Florida, and Indiana, and based on analysis of administrative data files. The National 
Evaluation of the Student Support Services Program, conducted by Westat (2010), was a 
longitudinal study that provided statistical comparisons on the academic achievements of 
program participants to those of a matched group of non-participants. Its quasi-experimental 
design involved 5,800 freshmen evenly divided into treatment and control groups, tracked for six 
years and evaluated using both quantitative and qualitative data. 


The two quasi-experimental studies found areas of effectiveness. For Talent Search it was in 
first-time applications for financial aid and initial postsecondary enrollment rates. The Student 
Support Services Program showed moderate but statistically significant effects based on 
differences in service levels. The more students participated in program services the greater the 
benefit on grade-point averages, credits earned, retention, and degree completion. The quasi- 
experimental design of these studies and their modest findings attracted little reaction, especially 
in comparison to that generated by the Upward Bound study, its use of random assignment, and 
its finding that the program had no major effects on college enrollment or completion. In spite of 
glaring weakness (Cahalan, 2009, Cahalan & Goodwin, 2014), its results were used to justify a 
request to eliminate funding for Upward Bound, Talent Search and GEAR UP, an OMB rating of 
“ineffective,” and a call for new strategies in the distribution of federal TRIO funds (Haskins & 
Rouse, 2013). 


From the beginning, the Upward Bound study did not go over well in the TRIO community, 
which voiced objections prior to the start of any work. However, program practitioners were 
given no opportunities to help define its scope or influence its implementation. The Department 
of Education required participation as a condition of funding, which made the study come across 
not as a search for effective practice but as a compliance investigation. This perception by 
practitioners had a significant and negative impact on the study’s implementation and the level of 
cooperation from the TRIO community. 


Program practitioners further objected to the way researchers sought parental approval for 
student participation in the study. As with programs, students had to agree to participate in the 
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study as a condition of acceptance into Upward Bound. The terms of acceptance required 
providing access to confidential information for a period of years. The contractor for the study 
explained these terms in letters that used dense language few parents of first-generation college 
students would understand. For this and other reasons, some universities prohibited their 
Upward Bound projects from participating in the study without an institutional review board 
examination for human subject violations. 


The study’s random assignment procedures required each project to solicit twice the number of 
applicants it needed to create a control group of program-eligible students. Students in the 
control group were to be denied services not just from Upward Bound but all college access 
programs (e.g. Talent Search). Program practitioners adamantly opposed this denial of services 
and expressed concern about the negative impact of this denial on the reputation of their 
programs. Since students in the control group were those willing to complete the extensive 
application process most Upward Bound projects require, they were also willing to look for other 
opportunities for college access participation once they were denied acceptance by random 
assignment procedures. Most of these students went on to participate in other college access 
programs (Upward Bound on other campuses, Talent Search, and GEAR UP) even though they 
remained in the Mathematica control group. This created statistical bias in favor of the control 


group. 


Cahalan (2009) and Cahalan and Goodwin (2014) issued pointed criticism of the Mathematica 
study of Upward Bound from the unique position of having been on the inside of the Department 
of Education as officials responsible for the study. They noted that “the 2009 (final) report was 
published over the objections of the ED career technical staff assigned to monitor the final 
contract, and after a ‘disapproval to publish’ rating in the formal review process from the Office 
of Postsecondary Education.” Their detailed review contained poignant criticism of the study’s 
overly ambitious design, its seriously flawed sample design, the atypical use of a single project 
to represent the largest stratum of institutional types, and the lack of balance between the 
treatment and control group. Based on these and other flaws, the Council for Opportunity in 
Education (COE) submitted to the Department of Education a formal Request for Correction of 
the Mathematica final report. The American Evaluation Association and American Educational 
Research Association signed a Statement of Concern based on this request. In spite of these 
criticisms, the Upward Bound evaluation study by Mathematica continues to be the basis for 
policy proposals and decisions in large part due to its use of random assignment. 


This history reveals why current emphasis on the use of rigorous research to validate program 
practice is disconcerting to TRIO administrators. Deciphering and using high-level research 
requires being familiar with an esoteric skill set that is very different from the compliance- 
focused work previously emphasized in federal grant administration. Based on experience with 
national evaluation studies, working with rigorous research can also be unrewarding and 
disruptive. Of the three evaluation studies commissioned by the Department of Education, only 
the Student Support Services evaluation identified effective practices that influenced how 
programs operate. Its finding that home-based and blended programs were related to improved 
student outcomes resulted in the increased use of these features in program models. However, 
the random assignment study of Upward Bound, which involved intrusive procedures that were 
disruptive to programs, students, and schools, produced no recommendations to improve 
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practice. With these experiences, it is understandable why TRIO practitioners would be cautious 
of a requirement to validate program practice with rigorous research. 


But these reasons to be cautious should not cause practitioners to overlook what can be gained 
from effective use of rigorous research. The problems that create the need for college access 
programs are complex and numerous, and solutions will be no less complicated. Well-structured 
and high-level research provides the best basis for examining this complexity and creating better 
outcomes for students. But this should not cause us to think that random assignment is an 
infallible panacea to be tolerated under all circumstances. As with any other tool, its 
effectiveness will result from appropriate use that starts with respect for the programs and clients 
it is intended to help. 


Judith M. Gueron (2000), a noted researcher with Manpower Demonstration Research 
Corporation and a proponent of random assignment studies, argues for the judicious use of social 
experiments and the careful interpretation of their results. She explains that these studies are 
administrative and ethical burdens and should not be used unless they are carefully developed, 
address the right question, and meet all ethical and legal standards. She lists other preconditions 
that require researchers to show that they can convince people that there is no easier way to get 
the answers, balance research ambition against operational reality, implement a truly random 
process, follow enough people for an adequate length of time to detect policy-relevant impacts, 
collect reliable data on an adequate number of outcomes, and assure that people get the right 
treatment. These difficult preconditions are more easily accomplished with the help of program 
practitioners. 


The National Center for the Study of Adult Learning and Literacy (NCSALL) has helpful ideas 
for involving practitioners in research. As discussed by Smith et al (2002), sustained 
interactivity among researchers and practitioners leads to greater research utilization. Providing 
practitioners with research information in an accessible form and arranging venues for 
exploration, reflection, and implementation encourage this interactivity. NCSALL uses its 
“Practitioner Dissemination and Research Network,” which is comprised of adult education 
teachers and administrators, to facilitate interactions between researchers and practitioners. 
These “practitioner leaders” assist researchers with data collection, review and interpretation of 
initial findings, and dissemination of results. 


All of the different players connected to college access and success programs can have a role in 
forging this type of cooperation between researchers and practitioners. The Department of 
Education can sponsor forums that bring together researchers and practitioners to hear each 
other’s interests and concerns. Through its ability to award grants, the Department can ask 
questions that encourage researchers and practitioners to work together to uncover answers. It 
can also encourage collaborations at the campus level by promoting partnerships between faculty 
researchers and program practitioners through its competitive preference process. 


On-campus collaboration can take place even without federal involvement. Demographic 
projections indicate that all sectors of higher education will see increased enrollment from 
student groups that are currently underserved. Colleges and universities will want to know more 
about how better to retain their Pell Grant recipients, improve their campus climate for 
underrepresented minorities, and create other opportunities for students who require assistance 
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from college access programs. By encouraging cooperation between faculty researchers and 
practitioners, existing campus-based college access programs can help individual colleges and 
universities develop interventions that are custom fitted for their unique situation. 


The efforts by professional associations to support and strengthen the connection between 
research and practice in college access and success program are significant. Noteworthy among 
these efforts is the 2003 project by the Social Science Research Council (SSRC) called 
"Transitions to College: From Theory to Practice." Sponsored by the Lumina Foundation, this 
project focuses on postsecondary transition and retention successes and failures for 
disadvantaged youth. It brought together scholars and practitioners from a variety of disciplines 
and methodological backgrounds to clarify what was known about the transition to college, 
identify gaps in this information, and create links to policy and practice. This work produced 
several field-based literature reviews of academic work in ten disciplines. It also led to a 2005 
publication by the SSRC entitled “Questions that matter: Setting the research agenda on access 
and success in postsecondary education.” Responding to this publication, the Pathways to 
College Network, Social Science Research Council, and the Institute for Higher Education 
Policy convened a 2007 “Questions that Matter” conference that involved researchers, 
practitioners, and policy makers in a discussion of future research on improving college access 
and success for underserved students. 


The collaboration involving the Association for the Study of Higher Education, the Council for 
Opportunity in Education, and the Pell Institute continues efforts by professional associations to 
bring researchers and practitioners together. Relatedly, the Association of Public and Land- 
Grant Universities and the American Association of State Colleges and Universities recently 
launched a project in which nearly 500 public colleges and universities committed to increase 
their numbers of college graduates, especially among underserved student populations. This 
work very much involves examining the practice of college access programs and creates 
opportunities for researchers and practitioners to do so through collaborations. 


Within K-12 education, the strength of teachers is recognized as the single most important factor 
in the education of poor and working class children. The same can be said for college access 
programs, which must address numerous concerns and issues that represent the full scope of 
consequences resulting from poverty and minority status. Program staff must have the particular 
knowledge and ability to address these concerns and to work within the various areas (financial 
aid, academic support, cultural enrichment, diversity training, mentoring, parent programming, 
faculty training, cross-campus collaborations, secondary school programming, evaluation, etc.) 
that collectively constitute an effective college access program. The qualification of staff is the 
primary basis for effective practice. The current emphasis on the use of rigorous research adds 
to these qualifications the ability to understand the structure of quality research, and the 
willingness to collaborate with researchers. A very important part of this responsibility is 
ensuring the highest regard for student concerns and well-being. Practitioners bring that special 
contribution to the table like no others. 


The kind of preparation required to administer effective college access and success programs 
should be formulated within the academic curriculum and recognized with academic credentials. 
Such a course of study should necessarily include the development of research and evaluation 
competencies. The graduate certificate program established by Colorado State University and 
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the Council for Opportunity in Education is noteworthy for providing current college access 
professionals with academic courses and meaningful qualifications directly related to their work 
in college access programs. As other colleges and universities adopt these academic programs, 
the better able college access practitioners will be to use and define effective research. 
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EVALUATION ERRORS: SIXTEEN LESSONS FOR 
RESEARCHERS AND PRACTITIONERS FROM THE 


MATHEMATICA UPWARD BOUND STUDY 


BY MARGARET CAHALAN 


Abstract 


The long running National 
Evaluation of Upward Bound (1992- 
2009) was considered a_ gold 
standard random assignment study 
designed to assess the average 
impact of the Upward Bound 
Program. After the study had been 
active for over a decade, and 
published several impact reports, a 
quality assurance review conducted 
by ED found serious sample design, 
treatment control group bias and 
analyses and reporting issues that 
were serious enough to impact the 
study conclusions. 


Using the Upward Bound 
evaluation as a case study, this 
essay presents 16 lessons learned 
for researchers and_ practitioners 
alike for future evaluations in 
designing, implementing, analyzing, 
and reporting results. 


ee 


The Trump administration’s proposals to continue 
funding only those programs that have “evidence of 
effectiveness” and the associated President’s 
FY2018 budget proposals to cut critical service 
programs, including TRIO and GEAR UP programs, 
have prompted me to prepare this essay. Mr. 
Trump’s threats and proposals are similar to those 
put forth by President Bush in his FY2005 and 
FY2006 budgets calling for zero funding of Upward 
Bound (UB), Upward Bound Math Science 
(UBMS), Talent Search (TS), and GEAR UP. These 
proposals were justified in President Bush’s budgets 
by the early and interim findings from _ the 
Mathematica Policy Research’s (Mathematica) 
National Evaluation of UB (Myers et.al. 1999; 
Myers et.al. 2004) that were later found to be 
erroneous (Cahalan 2009; Cahalan and Goodwin 
2014; Nathan 2013; Harris et.al. 2014). It was only 
after extensive practitioner lobbying in Congress in 
2006, by COE and other college opportunity groups 
that the programs were ultimately saved, albeit with 
continual decreases in number of students served 
due to the resulting level funding for over a decade. 
This level funding was only checked after 2016 with 
increases for TRIO in the 2017 and 2018 budgets. 


This essay is also written in response to the May 
2017 blog posting by Paul Decker, CEO of 
Mathematica Policy Research (Mathematica). Dr. 
Decker’s 2017 blog posting laments the lack of 
using research for program improvement rather than 
just for the “cutting block.” Few would disagree 
with Dr. Decker’s point that research should be used 


for program improvement; however, his Blog posting once again uses Mathematica’s erroneous 
UB evaluation results as an example of what research can contribute to program improvement. 
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The article contains no acknowledgement of the errors in the Mathematica reports. Nor does it 
argue for more valid and improved research. It also does not acknowledge the limited level of 
college preparation services available for low-income and first-generation students. Rather, the 
article argues for more funding for research of the same type as conducted by his company, 
Mathematica. Dr. Decker’s 2017 blog posting is a shorter, less aggressive form of his 2013 
APPAM Presidential address (Decker 2013) but again omits any mention of the glaring errors 
revealed in a Quality Assurance Review conducted by the technical monitors of the final of the 
three Mathematica Upward Bound evaluation contracts within the U.S. Department of Education 
(ED). 


As a lifelong contract researcher as well as a Technical Monitor for evaluation studies, I wish to 
share with you the specific evaluation lessons that I believe can be learned from my unsettling 
experience when I became employed within the U.S. Department of Education (ED) and served 
as the supervisor of a group responsible for the technical monitoring of a number of 
postsecondary evaluations. These studies included the final Mathematica contract for the long- 
running National Evaluation of Upward Bound. In this essay, I too, as did Dr. Decker, use the 
Mathematica Upward Bound study as an example. However, it is an example of the pitfalls that 
can happen to an evaluation when practitioner and stakeholder rights to a valid and transparent 
evaluation are not followed (see Joint Committee for Education Evaluation Standards, 2011). 
These lessons learned will hopefully inform both practitioners and researchers in the design, 
implementation, and analyses of future studies, in an era in which federal service programs are in 
serious jeopardy. 


The details of the errors in the Mathematica report, along with fully-documented positive and 
substantial impacts found when these errors are addressed using NCES statistical standards, are 
reported in documents published by the Council for Opportunity in Education (COE) in 2009 and 
2014 and in publications of researchers from the University of Wisconsin and Tulane University 
(Cahalan, 2009; Nathan, 2013; Cahalan and Goodwin, 2014; Harris, Nathan, Marksteiner, 2014). 
Links to these reports, and to COE’s 2012 formal Request for Correction can be found at the 
following site http://www. pellinstitute.org/publications- 
Upward_Bound_Compilation_of_Links.shtml. 


Background Overview of the Study. The National Evaluation of Upward Bound was hailed 
as a gold standard longitudinal random assignment study when it was begun in the early 1990s. 
The study was designed to follow about 3,000 middle school and early high school students 
interested in Upward Bound, who were randomly selected to be either given the opportunity for 
UB participation in the study years or not (treatment or control group). The design called for 
following both groups through at least six years after scheduled high school graduation. The 
RFP for the study called for a nationally-representative random assignment study design that 
would allow for estimating the average national impact of the Upward Bound program. The 
study design also included a descriptive Project Survey and several qualitative case studies of 
projects to gain an overall picture of UB services provided. Projects were also asked to keep 
detailed service records for study participants. 


Below, as I unfold the UB evaluation story, I list 16 interrelated lessons learned from the 
Mathematica National Evaluation of Upward Bound. While this essay is especially addressed to 
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researchers, it is also addressed to practitioners. I offer it in the hopes that the lessons learned 
will lead to new models for evaluation that emphasize respectful partnerships between 
practitioners and researchers. 


Sixteen Lessons Learned From the Mathematica Upward 
Bound National Evaluation Story 


1. 


Consult with All Stakeholders and Take Seriously the Feasibility Concerns of 
Practitioners Concerning What Will and Will Not be Feasible and Ethical. After 
the basic UB evaluation design had already been developed and at the time the study was 
first presented to the Upward Bound community, a number of concerns were raised by 
program practitioners who were required to participate. Practitioners expressed concern that 
the study would change the manner in which they interacted with potential participants, 
especially in recruitment, and also questioned whether the national study would yield valid 
and useful information for their differing local projects. A major concern had to do with the 
anticipated difficulty of actually implementing a random assignment study that would not 
result in IRB- prohibited denial of services concerns. There were also concerns expressed 
about the potential for control group contrast. Projects did not like the changes to the normal 
recruitment procedures they followed and specifically did not like the requirement that in the 
study years they needed to recruit at least twice the number of students than the anticipated 
openings and then purposely not give UB services to half of the students recruited. While 
project stakeholders were informed of the study requirements before implementation, they 
were not consulted on the basic study design. In large part these concerns by practitioners 
were ignored, and as will be seen below the accommodations made for stakeholders seem to 
have made the design even more problematic. 


Do Not Over-Promise and Sacrifice What Might be Possible for That Which Is 
Not Possible. In order to make the study seem more useful to stakeholders, as the study 
planning proceeded, the sample design evolved to attempt to allow for estimating not just 
national impacts but the differences in impacts between different types of projects hosted at 
different types of colleges (2- or 4-year); public or private; rural or urban; and serving 
participants of different race/ethnicities; and projects of different sizes. It was this attempt to 
have both national estimates and estimates for many different types of projects that resulted 
in a highly-stratified first stage sample of 67 projects drawn from 47 strata, some of which 
had only one project. Moreover, due to the need to respect projects’ existing patterns of 
recruitment, the final second stage student level sample was drawn from over 300 strata. In 
the final stage of inverse of the probability of selection weighting, one project, the single 
representative of a very large stratum, ended up carrying fully 26 percent of the weights. As 
we will see, as the UB evaluation story unfolds, the project sample design sacrificed possible 
overall estimates for a poor design incapable of producing either reliable national estimates 
or separate estimates for the various strata, and with serious unequal weighting. 


Do Not Assume an Error-Free Study Implementation of a Random Assignment 
Design. The contractor, Mathematica, is known for its expertise in “random assignment” 
and presented the procedures of the UB evaluation as the gold standard. In fact, there are 
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very few studies that attempt weighting using inverse of probability of selection for a multi- 
stages stratified sample with a random assignment design. As noted above, projects were 
instructed to carry out recruitment in a special way in the study period. This consisted in 
recruiting at least twice the number of students as anticipated openings to be on what was 
called a “waiting list” for Upward Bound. Only students who completed a detailed “baseline 
survey” were allowed to be eligible for the “waiting list” to apply for openings in the Upward 
Bound program over the two recruitment study years. However, projects implemented the 
request to submit at least twice the number of baseline surveys completed by potential 
participants as anticipated openings in different ways. Some administered the baseline 
surveys to large numbers of students who might have been in Talent Search (another TRIO 
pre-college program often beginning in middle school) or recruited from a whole 8" or 9" 
grade class in a low-income target school. Other projects submitted far fewer baseline 
surveys from students who might have been closer in time to being what might be considered 
“formal applicants.” As the weighting at each stage was the inverse of the probability of 
selection and the second stage weighting was done relative to the number of baseline surveys 
submitted rather than the actual number of openings, this difference in the number of 
baseline surveys relative to actual openings contributed further to the unequal weighting and 
balance issues. To feasibly implement the random assignment stage at the UB project level, 
the different projects were allowed to establish their own project-specific strata. For 
example, they could establish separate strata with different probabilities of selection by 
gender, or for the different target high schools served. This involved dividing the so-called 
“waiting list” by the additional stratification criteria —so in the final combined stage, as 
noted there were over 300 strata. Within a given project students thus had different 
probabilities of selection and second stage weights. The control group consisted of all 
baseline survey completers who were not randomly selected to be given the UB opportunity 
as openings developed within the project strata. Thus, the control group and treatment group 
did not have equal actual numbers within a given project. Weight adjustments were used to 
equalize the final stage weights so that the control group and treatment group in each project 
has an equal sum of weights. In addition, about 10 percent of the baseline survey submitters 
were selected with certainty to be in the UB program, due to previous commitments or group 
dynamics reasons and were removed from the study with their weights redistributed among 
the study treatment cases in their strata. 


Given Denial of Services Concerns in College Access, Do Not Assume that the 
Control Group in Practice Will Not Get Related Services. As openings for the UB 
program occurred over two summers, within the project strata, students were selected at 
random to be given the opportunity for program participation. Those not randomly selected 
became the control group. Subsequently, given student mobility and other factors, it was 
found through surveys and project records that about 25 percent of those selected to be given 
the UB opportunity did not report ever entering into Upward Bound, with some portion of 
these reporting they did not remember ever being given the opportunity. These students were 
considered in the UB Treatment Group in the “Intent to Treat (ITT)” analyses. Conversely, 
about 15 percent of the Control Group entered into Upward Bound or Upward Bound Math 
Science and were maintained in the Control Group. In addition and importantly, over a 
majority (60 percent) of the Control Group reported getting some form of other supplemental 
pre-college services in high school; most frequently, it was the less intensive Talent Search 
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(another TRIO program). Some UB practitioner staff reported that, in order not to be 
denying time-sensitive college access services, they actually tried to find alternative pre- 
college services for those in the control group who did not get randomly selected for the 
Upward Bound opportunity. 


. Reveal Any Issues Early—the Importance of Transparency. By the time I joined 
ED, three Mathematica reports had been published in 1996, 1999 and 2004. It was not until 
2005, soon after I joined ED and after the study had been active since 1992, that our ED- 
PPSS division director, Dr. Goodwin, and I were alerted to some problems with the UB 
study. This alert first came from a new Mathematica staff person working on analyses and 
report writing of the Fourth Follow-up, who subsequently left the company. The staff person 
was concerned after having found that the “no overall impact results” were being driven by 
only one project of the 67—with the unusually large weight, accounting for 26 percent of the 
total sum of the weights. It was further reported to us that this project, when considered 
alone, had an unusually large number of applicants (baseline surveys submitted) and also had 
very large negative impacts. These negative impacts were strong enough to impact the 
overall positive impacts found when this project was omitted. Because this project was 
supposedly representing a very large number of both projects and baseline survey completers 
(considered applicants), this flawed design meant that the outcomes of some students from 
this project “waiting list” carried weights that were 40 times those of the lowest weighted 
students from other projects. The first three Mathematica UB evaluation reports, published 
over almost a 10-year period, did not reveal these serious sample design issues. 


. Investigate Data Results that Defy Logic. At the time, as ED Technical Monitors, we 
asked Mathematica to investigate why the heavily weighted project, when considered 
individually was having such large negative impacts on college going so that we could 
understand what the project was “doing wrong.” We could understand “no impact” results, 
but very large significant negative impacts did not seem plausible. Unfortunately, we did not 
get a response from Mathematica on this request. Instead we received a draft of the Fourth 
Follow-up report that, similar to the three previous Mathematica reports, did not reveal the 
unequal weighting issues, or that the lack of impact results were being driven by only one 
project. In the course of reviewing the Fourth Follow Up report as Technical Monitors, we 
were also struck by the very large positive significant impacts found for the sub-group of 
students that were classified as being of high academic risk at the start of the study. The 
results reported that there was an over 20 percentage point significant positive UB impact on 
college enrollment based on differences between the academically at-risk group treatment 
and the at-risk control group in college entrance. At the same time, it was reported that there 
were no overall impacts. We were also struck that, contrary to NCES statistical standards, 
the outcome measures had not been standardized by high school graduation year for a sample 
that spanned five years of scheduled high school graduation dates (given that the study 
recruited students from 8" grade to 11 grade and over two summer program rounds). When 
we asked Mathematica to standardize the outcome measures, Mathematica responded that 
they did not need to do this standardization given that there had been a random assignment 
that should have resulted in equal division between the treatment and control group on this 
distribution. However, ED monitoring staff balance checks later found that the control group 
was on average in a higher grade at the start of the study than the treatment group and that 
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when we standardized outcomes that the overall conclusions changed—even with the outlier 
heavily weighted project with 26 percent of the weight and negative individual impacts. 


. Pay Attention to Quality Review Results. Given the lack of response of Mathematica 
to our concerns, in 2006, as Technical Monitors, we requested the data files from 
Mathematica and began an internal ED-PPSS Quality Assurance (QA) review of all data files 
from the study. At first, we were primarily concerned that the study was relying solely on 
survey results and that the non-response issues were aggravated by the unequal weighting. 
Knowing that the administrative records from the federal student aid application and award 
data was available for the entire sample, we arranged to have the sample matched to the 
federal aid files and merged on to the UB study files for analyses. We also consulted other 
external statistical contractors and sent the Upward Bound data files to RTI, which held the 
ED-PPSS Statistical Technical Assistance contract at the time, for review and replication of 
our internal findings. The ED staff internal and RTI external quality reviews revealed that 
the study had serious sampling and non-sampling error issues that had not been revealed by 
Mathematica over the course of the more than a decade of contracts and after the publication 
of a major impact report in 2004. On the basis of using the federal aid files, ED-PPSS 
internal analyses, replicated by RTI, began to find positive overall impacts on postsecondary 
outcomes. 


. Make Sure there is an Actual Balance between the Treatment and Control 


Group through Balance Checks. The balance analysis done as part of the Quality 
Assurance Checks revealed that the negative impacts in the outlier weighted project reflected 
a failure of the implementation of the multi-stage random assignment rather than negative 
treatment impacts. The reason for random assignment is to ensure an equivalent treatment 
and control group on factors associated with the outcome or a balanced treatment and control 
group. The negative individual site impacts observed in the project carrying 26 percent of 
the weights were due not to program practices but to large differences between the treatment 
and control group on academic risk, educational expectations, and grade at baseline within 
this project. In what can only be a failed implementation of the random assignment, 80 
percent of the high academic risk students from this project were in the treatment group and 
20 percent in the control group. The control group from this project was also on average in a 
higher grade in high school at the start of the study. In fact, the treatment group from this 
project was contributing a large percentage of the overall weight for students of high 
academic risk in the overall sample, while the control group from this project was 
contributing a larger proportion of the students that were academically talented and had 
advanced degree expectations. Among the total sample, an average of 36 percent expected an 
advanced degree, compared with 56 percent in the control group from the outlier project, and 
15 percent of the treatment group from this project. This explained the large positive impacts 
for the high academic risk sub-group in the analyses. When only those with high academic 
risk in the sample were considered, the high-performing, highly-weighted members of the 
control group from the unbalanced project was not included. Within the high academic risk 
sub-group, there was by definition a more balanced treatment and control group. Hence the 
very high UB sub-group positive results for academically at-risk students emerged (Cahalan 
2009). 
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The large weights among this unbalanced project combined with the lack of treatment and 
control group balance led to an unacknowledged and uncontrolled bias in favor of the control 
group among the overall sample forming the basis of all of the published Mathematica 
national estimates. This issue combined with the lack of standardization of outcome 
measures by high school graduation year and other analysis and reporting errors were serious 
enough to impact the overall conclusions Mathematica reported concerning the Upward 
Bound project. Moreover, (contrary to what Mathematica had reported in 2004, continued to 
report in 2009, and was repeated by Dr. Decker in his 2013 APPAM Presidential Address) 
strong positive impacts for the overall sample, including the bias-introducing outlier project, 
were found for UB when these errors were addressed by such common statistical standards 
requirements as standardizing the postsecondary outcome measures by scheduled high school 
graduation date. 


. Check the First-Stage Sample for Atypical Cases that Cannot Represent Their 

Strata before Proceeding and Check the Eligibility of those Considered 
“Applicants”. To protect confidentiality, the data files delivered to ED in 2006 and 2007 
for QA review did not reveal the identity of the UB projects in the sample. In the course of 
reviewing the study, Dr. James Chromy, the RTI statistician we consulted, asked us to 
request a copy of the sampling frame from Mathematica in order to gain a better 
understanding of the sample design characteristics. We made this request of Mathematica in 
early 2007; however, Mathematica indicated that they did not have an electronic version of 
the sampling frame and could not locate the paper version for nine months. Consequently, 
this frame was not delivered to ED until a few weeks before the final contract ended in late 
2007. Until this time, no one at ED or RTI was aware of the identity of any of the study 
projects. A review of the sampling frame in late 2007, (something that should have been 
done by Mathematica in the 1990s, before the study began), revealed significant facts that 
explained the strangeness of the results we were observing. In a flawed design, the project 
carrying 26 percent of the weight was selected to be the single project representing the UB 
projects hosted at 4-year public, urban, non-majority Hispanic institutions, and average- sized 
UB project. In other words, this single project was representing a lot of UB projects. This, 
combined with the large number of baseline surveys submitted, resulted in its very large 
outlier weights. To compound this problem, when we researched this project’s 
characteristics, we found that unfortunately it was very atypical for the 4-year public stratum 
for which it was supposedly the sole representative. It was housed at a former private junior 
college serving minority youth from minority vocational high schools that historically 
awarded postsecondary trade certificates. The junior college had been taken over by a large 
public city college system and hence its 4-year public formal classification in IPEDS. This 
accounted for the fact that the only positive overall impacts that Mathematica reported were 
for the award of certificates. Dr. Chromy and the other external reviewers noted that having 
only one atypical project representing the largest stratum of 4-year bachelor’s degree 
granting projects could not produce robust estimates for that stratum or for the whole, and 
that this was especially problematic for any estimates of bachelor’s degree attainment. 
Checks should have been done of the design, and on the actual randomly drawn first stage 
project sample prior to beginning the study to ensure that the cases drawn were not “atypical” 
for the strata they were representing. In this case, Dr. Chromy noted that the sample should 
have been re-drawn prior to beginning the study. 
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10. 


11. 


The representation issues of this heavily-weighted project were compounded by the actual 
non-equivalence of the treatment and control group in this project. For unclear reasons, the 
implementation of the random assignment in this outlier project appears to have been flawed. 
The Treatment Group resembled on average the types of vocational students historically 
served by this atypical UB program, while the control group on average resembled the types 
of students most frequently served by Upward Bound Math Science (UBMS)--in a higher 
grade at entrance, more academically talented, and with higher advanced degree expectations 
at baseline. Indeed it is unclear whether these control group members were actually 
completing the “baseline survey” because they were interested in the sampled particular 
Regular Upward Bound program with ties to vocational certificates, or actually had more of 
an interest in a new UBMS program in the region, (not in the sample) in which some of the 
control group from this project reported participating. These “baseline survey completers” 
when weighted constituted over one-quarter of overall sample control group and were 
probably not serious applicants for the (at that time) vocational certificate-focused UB 
project. 


Understand that Response and Coverage Issues Are Important; Follow NCES 
Standards for Response and Coverage; Note that Unequal Weighting Can Make 
the Non-Response Issues More Significant. One of the most problematic aspects of 
the Mathematica analyses in the final oh follow-up 2009 report was the ignoring of NCES 
coverage standards and the use of the National Student Clearinghouse (NSC) at a time when 
the coverage by institutions reporting to the NSC was about 27 percent for enrollment and 
when NSC was not yet collecting degree information. This coverage issue was especially 
problematic for less than bachelor’s degree-granting postsecondary institutions and was far 
less than would meet NCES standards for coverage. In the 5" and final report, survey non 
responders who were not found on the NSC data base were assumed to have not entered into 
postsecondary education or not to have obtained a postsecondary credential. This issue was 
aggravated by the unequal weights. For example, some sample members had weights of 158, 
whereas the lowest-weighted sample member carried a weight of 4. This was especially 
problematic as the heavily-weighted outlier project was not reporting to the NSC in the most 
applicable period when the students would be entering and completing postsecondary 
credentials. 


Make Sure that the Conduct of the Study Is Integrated. The Mathematica formal 
structure at the time provided for separate groups responsible for various aspects of a study, 
the statistical staff, data collection staff, qualitative case study staff, and econometric staff. 
Given the very long time period covered by the study there was also considerable turnover of 
staff. It is therefore not known if the project leadership or analysts of the various groups 
knew that the highly-weighted, vocationally-focused UB project was supposed to be the only 
representative of a large 4-year public hosted stratum, or that the treatment and control group 
were so unbalanced in this site. It is known that the separate significant “negative” impacts 
of this site were observed by 2005. We also do not know how many of the issues revealed in 
the ED-QA review were known to junior and mid-level Mathematica staff working on the 
study, at various times over the 15-year study. It is not known why the study error issues and 
the unequal weighting were not included in the reports through the third follow up in 2004. 
It was not until 2005/06 that these issues were called to ED’s attention and the full extent of 
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the issues was not revealed at this time. The seriousness of the errors became known by ED 
only gradually as ED-PPSS and external reviewers examined the data files and the sampling 
frame for themselves between 2006 and 2008. The federal aid matches were done in 2006 
and as noted the sampling frame was not delivered to ED-PPSS until late 2007 when the 
contract was essentially over. It was not until the contract was officially over in 2007- 2008 
that ED-PPSS internal staff worked to standardize results by high school graduation year and 
with this mitigation found substantial and significant positive impacts with and without the 
outlier project in college entrance, award of federal aid, award of any postsecondary degree, 
but (as might be expected given the unequal composition of the treatment and control group) 
not for bachelor’s degree receipt. The outlier project was not representative of its 4-year 
stratum and had seriously unequal treatment and control groups that introduced bias in favor 
of the control group for bachelor’s degree receipt that given the large weights was large 
enough to override the clear large positive impacts on bachelor’s receipt when the 
unrepresentative and bias introducing project was removed from the analysis. When the 
outlier project was included based on non-response adjusted survey results there were large 
impacts on the award of any postsecondary credential due to the large impacts for certificate 
receipt. 


Respect Stakeholder Rights to a Transparent Evaluation with Warranted 
Conclusions. The results of the ED-PPSS technical monitor’s analyses were fully shared 
with Mathematica over the period of 2006 to 2009 as they became known—however, they 
were repeatedly disregarded by Mathematica project staff. Mathematica’s leadership in 
formal letters accused the ED-PPSS technical monitors as overstepping and acting as 
advocates for Upward Bound. Memos to Mathematica concerning the data quality review 
results and errors identified written by myself and also by Dr. David Goodwin, the original 
UB study project monitor and the head of the ED-PPSS Division responsible for the study in 
2008, were repeatedly disregarded by Mathematica project staff and leadership. The 
repeated failure of Mathematica to reveal the issues with the study to ED and the 
stakeholders over a period of more than a decade, and their failure, to this current time in 
2018 to acknowledge the results of the PPSS QA analyses finding positive impacts, 
constitutes a serious negligence of the trust of that must be present in all evaluation contracts. 
This behavior also constituted a serious violation of trust to the Upward Bound and TRIO 
community. Stakeholders to evaluations have a right to a transparent and ethical evaluation 
in which only warranted conclusions are put forth concerning their effectiveness. 


13. Acknowledge All Findings, Especially if They Have Different Conclusions in 


Transparent Reporting. The statistically significant and educationally meaningful 
positive results PPSS internal staff and RTI external quality review consultants had found, 
and which had been fully shared with Mathematica by 2008, went unacknowledged in the 
2009 published Mathematica final report. When ED staff members applied NCES standards 
to the analyses (such as standardizing outcome measures, and respecting coverage standards 
for use of the NSC data) significant and substantial Intent to Treat (ITT) and Treatment on 
the Treated (TOT) positive impacts were observed for the entire sample, with larger impacts 
when the problematic bias introducing non-equivalent treatment and control group project 
was excluded. The final Mathematica report conclusions ignored their own analyses 
(presented in an appendix to their final report) that found a substantial and significant 12 
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percentage point-effect size for the award of any postsecondary degree based on survey data 
adjusted for non-response by the end of the study for the entire sample. This finding was 
found by both the ED-PPSS analyses and the Mathematica analyses, but in Mathematica’s 
case it was kept buried in an appendix to the Mathematica report and ignored in the text 
discussion in the disseminated Mathematica final report conclusions. The often-quoted 
Mathematica conclusion (Sefter et.al. 2009; Haskins and Rouse 2013, Decker 2013) that UB 
had no discernible impact on postsecondary entrance or degrees earned is clearly incorrect. 
The ED-PPSS analyses also found that omitting the atypical non-representative UB sampled 
project with an extreme weights and an unbalanced treatment and control group, the UB 
participants were 3.3 times as likely to obtain a bachelor’s degree in six years compared with 
study participants who received no supplemental college access services (Cahalan and 
Goodwin 2014). 


Don’t Underestimate the Influence of Political Appointees in the Review Process 
and in Deciding Which Reports Get Published. The final Mathematica report was 
published by ED in the last week of the Bush Administration (January 2009) only upon direct 
orders from the departing political appointee staff. It was published over a year after the 
final contract formally ended in late 2007, and over the clear written formal objections of 
ED-Policy and Programming Studies Services (PPSS) career technical monitoring staff. It 
was also published after a formal “disapproval to publish” rating in the final Executive 
Secretary’s (Ex. Sec) review from the Office of Postsecondary Education (OPE), out of 
whose appropriation the $14 million dollar study was funded. Its actual formal classification 
in the ED review process at the time it was published was “Returned to ED-PPSS for re- 
write.” ED’s formal publication review procedures were circumvented, and ED-PPSS was 
ordered in early January 2009 to get the report out by the end of the Bush Administration. 
While the ED-PPSS Technical Monitoring staff was not given details of the final 
negotiations with Mathematica, which were conducted in secret by the departing political 
staff, the reports were published with reported acquiescence and facilitation of the Institute 
for Education Sciences (IES). At the time, IES had a former Mathematica Vice President in 
a leadership position. In fact, in defiance of Mathematica’s formal contractual agreements 
with ED, Mathematica had already published the Mathematica draft UB Evaluation final 
report on its own website in December 2008. After the Mathematica report was officially 
published by ED in 2009, PPSS attempted to get the results of their Quality Assurance 
Review also published by ED; however the publication of the QA results was blocked. 
Subsequently, ED-PPSS leadership gave permission to have COE publish the PPSS QA 
results in late 2009 (Cahalan 2009). 


In a further “suppression of transparency” after both Dr. Goodwin (the division director of 
the unit responsible for the study) and I had left the Department of Education, the UB study 
files were ordered withdrawn from availability for restricted release to other researchers. The 
files had been stripped of identifiers under a separate contract by RTI and made available 
under restricted release to other researchers in early 2011. The order to withdraw the data 
files came from the same former Mathematica senior staff person who had by this time 
become in charge of PPSS. This withdrawal of the data files was ordered in 2012, after 
having been released to only two sets of external researchers (University of Wisconsin and 
COE’s Pell Institute). The researchers from the University of Wisconsin subsequently 
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replicated the internal ED staff quality assurance results, also finding positive results for the 
Upward Bound programs. 


The history of the Upward Bound evaluation shows how difficult it is for official Technical 
Monitors to do anything but rubber stamp findings by a respected, so-called “objective 
independent contractor.” The Bush Administration’s political appointees were looking for 
evidence that the federal TRIO programs were ineffective. In fact, the Assistant Secretary’s 
representative to whom PPSS reported openly joked in regularly scheduled meetings with our 
PPSS group that my supervisors should stop me from working on the UB project. After 
raising concerns about the Upward Bound study, I experienced the classic responses to a 
“whistleblower” within the ED bureaucracy. As noted, after a review of the UB data, Dr. 
Goodwin, the original UB study Technical Monitor in the 1990s and then the ED-PPSS 
Division Director, also wrote a detailed memo to Mathematica leadership expressing his 
view that the UB study was “seriously flawed.” However, he too was ignored by 
Mathematica, IES, and the ED political leadership. 


In the Current Climate of Government, if a Study Can Be Used to Cut Services, It 
Will Be Used. Among some political operatives and researchers, it is the working 
hypothesis that government programs of the civil rights and War on Poverty era do not work, 
do not target the right persons, or at best are not worth the resources put into them. In 2011, 
Grover T. Whitehurst, former IES director, testified to Congress that Upward Bound and 
Head Start and similar programs had not been shown to be effective. The often-quoted 
Haskins and Rouse (2013) Brookings policy brief based on the Mathematica findings 
generalized to calling most existing college access programs “ineffective” and called for 
radical restructuring of the service programs into research demonstration evaluation projects. 
In November 2013, Paul Decker, President of Mathematica, went so far in his APPAM 
Presidential speech as to call those he labeled the “Youth Advocacy Community” as being 
guilty of “misdemeanors” and “felonies” for their legitimate expressions of concern relative 
to the proposed design for a new Upward Bound evaluation and for their successful lobbying 
in Congress for language in the Higher Education Act of 2008 to require that future federal 
evaluation studies in TRIO meet IRB requirements. More recently again in 2017, Paul 
Decker arrogantly used the Upward Bound results to argue that college access programs have 
not shown “clear evidence of effectiveness.” In a circular process, he shamelessly cites the 
Haskins and Rouse proposals, which were justified by using the erroneous Mathematica 
flawed UB results, to argue for more research of the type conducted by his company on how 
to make the so-called “ineffectual” programs more effective. 


Correct Mistakes, Do Not Cover Them Up. Several people have asked me why 
Mathematica, a well-respected firm—holder of the What Works Clearinghouse (WWC) 
contract--would be so unwilling to admit mistakes. I believe that part of the issue is that in 
the very competitive contract acquisition system it is very easy to overpromise and also to 
ignore basic questions about the validity and robustness of the proposed design to address the 
complex questions posed by the government. Once contractors have published high-profile 
study results, the perceived need to protect their own organizational reputation can make it 
very difficult to admit to themselves and the public, that they may have made mistakes. In 
this case, practitioners and other stakeholders and ED-PPSS technical monitoring staff who 
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raised questions were portrayed by Mathematica as being advocates who were anti-data, anti- 
random assignment, and ignorant of research methods. 


Hopefully the “Upward Bound evaluation story” is an “outlier” in the history of evaluation 
research. It was a “perfect storm” example of a complex, long-running evaluation that “got it 
wrong” combined with political actors bent on zero funding the program and who welcomed any 
findings that would support the budget policies they were advocating. The Mathematica 
contractor, who also was the WWC contractor, refused to consider that they might have made 
mistakes and the leadership and staff “dug in their heals” against acknowledging the findings of 
the Quality Assurance Review. The contractor, with a strong reputation, also had powerful 
former staff allies in leadership positions in IES and later in PPSS. Given the current Trump 
Administration’s proposals to cut program services, we ask the question, will there be more cases 
such as the Upward Bound evaluation? How can we avoid the mistakes of the Upward Bound 
evaluation? How can we foster feasible, ethical, transparent, accurate, and useful evaluations 
that are not harmful to those we are serving? Who is looking out for the rights of the 
stakeholders to the evaluations, especially in an era in which social service programs are targets 
for budget cutting or elimination? 
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CHAPTER II 


HOW TO MAKE 
COLLABORATION WORK 


RELATIONSHIP AND TRUST-BUILDING BETWEEN 
RESEARCHERS AND PRACTITIONERS: TOWARD 


EDUCATIONAL EQUITY FOR UNDERSERVED 
POPULATIONS aysupy marquez KIYAMA 


Introduction 


Much has been written about the importance of 
collaborative efforts between faculty and student affairs 
practitioners (e.g. Banta & Kuh, 1998; Dale & Drake, 
2005; Kezar, 2003; Pace, Blumreich, & Merkle, 2006). 
Nonetheless, the topic remains at the forefront of 
institutional efforts, scholarly publications (see Dale & 
Drake, 2005; Magolda, 2005) and higher education and 
student affairs courses (e.g. Espino, 2012). Scholars 
push collaborations that are systemic and 
institutionalized in an effort to increase student success 
and retention (Dale & Drake, 2005) and research 
indicates that such collaborations do in fact, enhance “the 
quality of life for students” (Magolda, 2005, p. 16). In 
this essay, I focus specifically on the collaborative role 
between researchers and practitioners, with particular 
efforts that address relationship building to enhance 
educational equity for underserved populations. 
Researchers and practitioners often work towards 
common goals when considering access and success 
efforts for traditionally underserved populations 
(Kiyama, Lee, Rhoades, 2012), yet they remain 
positioned in dichotomous roles, separated by 
organizational structures (Pace, Blumreich, & Merkle, 
2006), with faculty responsible for student learning and 
research, and practitioners responsible for student 
support (Dale & Drake, 2005). 


Challenges Facing Collaboration 


Abstract 


Researchers and practitioners 
often work towards common 
goals when considering access 
and success’ efforts for 
traditionally underserved 
populations. Yet, 
collaborations across these 
roles are rare and when they 
do occur, can be met with 
organizational, political, 
financial, and __ ideological 
challenges. This essay will 
explore some of _ these 
challenges and focus 
particular attention on the 
benefits of such collaborations 
when relationships are built on 
trust and reciprocity. 


Authentic collaborations between researchers and practitioners, beyond infrequent meetings and 
committee work, are rare and when they do occur, can be met with organizational, political, 
financial, and ideological challenges. These challenges are helpful to understand within an 
organizational culture framework (Lee & Kiyama, 2005). Organizational culture is typically 
characterized as an interconnected web (Geertz, 1973) with shared norms, values, and beliefs 
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that are often taken for granted (Bolman & Deal, 1997; Deal & Kennedy, 1983; Lee, 2007; Lee 
& Kiyama, 2005; Morgan, 1986). Therefore, culture can also be understood differently by the 
various groups within an organization, in this case — researchers and practitioners (Lee & 
Kiyama, 2005). Differing organizational structures, or organizational fragmentation can be a 
barrier to collaboration, as specialization among faculty and staff and a history of separation 
between various units is likely to exist (Kezar, 2006). Organizational structure can also create 
competition for resources (financial and human resources) and differences in leadership 
ideologies and expectations (Pace, Blumreich, & Merkle, 2006). Because these subcultures 
within organizations establish different values, skills, and separation (physically and 
hierarchically), challenges in cross-departmental collaborations exist (Dale & Drake, 2005; 
Kezar, 2003; Lee & Kiyama, 2005). The separation that results from different departmental and 
disciplinary boundaries can undermine potential collaborations (Love, Kuh, MacKay, & Hardy, 
1993). 


Researcher and practitioner partnerships are also met with challenges around management 
structures to support the collaboration, group and individual dynamics, a balance between 
building community or professional purpose, and the knowledge base of those assuming 
leadership for the partnership (Amey & Brown, 2005; Bernacchio, Ross, Washburn, Whitney, & 
Wood, 2007). Limited time from faculty and staff, unclear goals for the collaboration, and a lack 
of senior leadership support have also been noted as barriers to successful collaborations (Kezar, 
2001). Even with the organizational and cultural challenges present when attempting to establish 
partnerships between researchers and practitioners, successful collaborations do exist. 


Developing the Collaboration 


The literature on researcher and practitioner collaborations often identifies pragmatic tips or 
strategies focused on organizational culture and leadership. For example, Whitt and colleagues 
(2008) analyzed collaborations across 18 different institutions and suggest the following 
recommendations: partnerships should advance the institutional mission; demonstrate learning- 
oriented environments; promote relationship building; recognize, understand, and value 
institutional culture; value and implement assessment; be good stewards of resources, and 
promote diverse opportunities for leadership. Specific to the focus of this essay, when 
considering relationship building, Whitt et al (2008) state that “effective partnerships grow out of 
existing relationships between and among academic and student affairs professionals” (p. 241). 
These relationships are built on shared values and often cross the organizational boundaries often 
noted as challenges. Their findings suggest that relationships are one key to the success of these 
collaborations (Whitt et al., 2008). 


Other factors or “mediating tools” of collaboration include shared understanding of institutional 
missions and organizational culture (Amey & Brown, 2005; Kezar, 2005). For example, 
institution missions or philosophies are important to integrate into the culture of the institution 
through events like public forums or ongoing initiatives like student engagement (Kezar, 2005; 
Kezar, 2006). Organizational structures can be redesigned to better support collaborations 
between researchers and practitioners (Kezar, 2005; Kezar, 2006). Kezar (2003) specifically 
notes the importance of structural strategies like formal organizational rules and planning 
processes, and cultural strategies like dialogue and common vision. Shared dialogue and 
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language and ritualizing norms appear regularly as important tools to developing 
interdisciplinary and cross-departmental collaborations (Amey & Brown, 2005; Kezar, 2003). 


There is often an assumption that collaborations are best initiated from a top-down approach or 
at least with senior leadership support in an effort to institutionalize policies and practices that 
sustain the outcomes desired from such collaborations (Harris, 2010; Kezar, 2001; Kiyama, Lee, 
& Rhoades, 2012). However, as Whitt et al. (2008) note, some of the most effective partnerships 
develop not from a top-down approach, but from preexisting shared values. This is discussed 
further in the next section. 


Partnership Principles Informed by Community, Activism, and 
Agency 


Collaborations can be built upon relational networks that are activist in nature, cut across 
different department cultures and administrative silos, and work towards organizational, equity- 
based change (Kiyama, Lee, & Rhoades, 2012). Kiyama and colleagues term these networks 
“critical agency networks.” The model of critical agency networks was suggested after an 
extensive study of faculty, researchers, student affairs practitioners, and academic administrators 
coming together for the creation and facilitation of a college outreach program serving low- 
income and families of color. The collaboration offered few tangible rewards; that is, no 
monetary compensation was offered and many faculty members did not even list the 
participation as a “service.” The collaboration was an outgrowth of previously formed 
relationships, developed out of common social justice and equity-based values (Kiyama, Lee, 
Rhoades, 2012). 


Another example of a critical agency network coming together is seen in Harris and Kiyama’s 
(2015) partnership with a local school district. The project was initiated by the president of a 
local community organization serving Latina/o students and families. The president issued a call 
to action after consistently high drop-out rates for Latina/o students in the local school district. 
What followed was the assembling of school district personnel, higher education professionals, 
researchers, faculty, students, and parents who engaged in relationship building to develop the 
trust necessary to collectively carry out the project. While the project originated as a research 
effort, it has led to both programmatic and curricular efforts as well. The goal of this project 
continues to be structural changes leading to opportunities for Latina/o students as they transition 
through high school and into higher education (Harris & Kiyama, 2015). 


These particular examples are reflective of community-driven or grassroots collaborations 
(Kiyama, Lee, & Rhoades, 2012). These grassroots efforts are illustrative of bottom-up 
networks organizing around a common issue (Kezar & Lester, 2009). Important in Kezar and 
Lester’s (2005) research is the role that virtual and external networks (sometimes within 
community contexts) can play in establishing these grassroots collaborations that originate on 
college campuses. I turn briefly to a discussion on community-engaged work, the principles of 
which not only highlight the resources found in external and community networks, but offer 
guidance for establishing authentic relationships across institutional contexts as well. 


Community-engaged collaborations, known often as community-based research, are defined as 
“a partnership of students, faculty, and community members who collaboratively engage in 
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research with the purpose of solving a pressing community problem or effecting social change” 
(Strand, Marullo, Cutforth, Stoecker, & Donohue, 2003, p. 3). If we understand “community” 
broadly to also encompass partnerships between researchers and practitioners, we see that the 
principles of community-based research are quite useful. Specifically, partners work together to 
design and implement projects, community (partner) knowledge is valued, and works toward a 
shared understanding to address issues (Israel, Eng, Schulz, & Parker, 2005; Polanyi & 
Cockburn, 2003). Key to these partnerships is the component of “with,” which encompasses 
developing a collective understanding at each stage of the project, rather than one partner taking 
over the power role as the leader (Israel et al., 2005; Polanyi & Cockburn, 2003; Strand et al., 
2003). Magolda (2005) also suggests that researcher and practitioner collaborations are possible 
and beneficial when there is shared commitment and power between partners, rather than 
researchers or faculty assuming the lead role. Shared leadership and power thus recognizes the 
multiple organizational frames and various forms of knowledge that each constituent brings to 
the partnership (Kiyama, Lee, & Rhoades, 2012; Magolda, 2005). 


Interestingly, Whitt et al., (2008) begin their article with a quote from the American Association 
of Higher Education which states, “People collaborate when the job they face is too big, too 
urgent, or requires too much knowledge from one person or group to do alone.” (p. 235). 
However, my own research and the examples shared above suggest otherwise. Underlying all of 
these examples and principles offered above are common threads, focused on issues of social 
justice and equity that bring researchers and practitioners to collaborative partnerships with long- 
term goals of systemic change. They are not brought together because the job they face is too 
big for one particular person; they are brought together because of a shared notion of critical 
agency--put another way, a shared commitment to advancing equitable opportunities for 
underserved students and communities. 


Values of Relationship and Trust Building 


Thus, in drawing from ideas embedded within developing critical agency networks (Kiyama, 
Lee, & Rhoades, 2012) and principles for community-engaged work (Strand et al., 2003), the 
following are noted as important values of partnerships based on relationship and trust-building 
between researchers and practitioners. 


1. Partnerships develop out of previously formed relationships. As was 
evident in both the Whitt et al., (2008) study and the Kiyama, Lee, and Rhoades 
(2012) study, authentic partnerships between researchers and practitioners are likely 
to develop when previously formed relationships already exist. Often, these 
relationships are built upon the values noted below. 


2. Dialogues, open and _ frequent communication, and _ authentic 
conversations. These conversations should include commitment to shared 
understanding about the sociohistorical nuances of the issue at hand, the sociocultural 
and moral discourses that inform the issue, and the organizational politics each 
partner faces in further addressing the issue (Harris & Kiyama, 2015; Magolda, 
2005). 
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3. Reciprocal relationships. Reciprocity and mutually beneficial social exchanges 
within relationship building has been noted as a key component within social capital 
frameworks (Coleman, 1988), sociocultural frameworks like funds of knowledge 
(Moll, Amanti, Neff, & Gonzalez, 1992), and within community-engaged work 
(Strand et al., 2003). It is not surprising then, to suggest that reciprocal relationships 
will also strengthen the partnerships formed between researchers and practitioners 
(Magolda, 2005), particularly when addressing issues of educational inequities. 


4. Relationships built on critical agency. Relationships that are built upon critical 
agency (Baez, 2000) share agendas that are activist in nature with movement towards 
a common social justice and equity-based goal. These partnerships are often created 
with and alongside members of the diverse communities such initiatives are meant to 
serve. Thus, these relationships can include researchers and practitioners internal to 
the institution and community members (i.e. students, families, non-profit 
organizations) that are also external to the institution (Kezar, 2006). 


5. Community-driven or grassroots efforts. Building off of the fourth point 
above, collaborations can start with members of the community. Grassroots efforts 
then drive the relationship and collaborative development by identifying and 
informing the efforts. (Israel et al., 2005; Kezar & Lester, 2009; Polanyi & Cockburn, 
2003; Strand et al., 2003). 


It may appear that divisive silos orchestrated by organizational departments, values, and 
disciplines hinder partnerships between researchers and practitioners. Yet, perhaps one of the 
most important points of this essay is that authentic relationships between multiple constituents, 
built on shared values and trust, can overcome such organizational boundaries in an effort to 
establish equitable opportunities for students. In an academic environment where results, 
numbers, and productivity prevails; it is necessary to recognize that systemic change takes time. 
Investing in the partnerships that will lead to such change remain worth the effort. 
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BUILDING TRUST BETWEEN RESEARCHERS AND 


PRACTITIONERS THROUGH RESEARCH 
(ee) LLABO RATIONS BY KRISTAN VENEGAS 


Abstract 


The goal of this essay is to 
explore ways that researchers 
and practitioners can work 
together to build trust in 
research collaborations. The 
essay considers building trust 
through research in_ three 
ways: (1) how researchers can 
work with practitioners to 
develop meaningful research 
designs; (2) how researchers 
and practitioners can 
collaborate around the 
technical aspects of research, 
including getting access to 
quantitative datasets and 
getting into qualitative 
fieldwork settings, and (3) how 
practitioners can work with 
researchers to make strategic 
improvements based — on 
research findings. 


a 


Introduction 


Trust plays a key role in research collaborations. As 
Kramer and Tyler (1996) suggest, trust impacts the key 
social functions including cooperation, coordination, and 
performance. Their work focuses on developing trust 
within organizations. In many ways, developing a 
collaborative research partnership is like developing a 
new organization. Informal and formal cultures are 
developed. Structures for leadership and processes for 
communication, the creation of mutual goals and 
implementation are established. Bolman and Deal’s 
(2011) descriptions of organizational culture can apply 
here. Like organizational cultures, research 
collaborations can take on one or more of the four 
organizational frames that Bolman and Deal have 
identified --- symbolic, political, human resource, and 
structural organizational frames. A _— symbolic 
organization is one that functions largely based on the 
prevalence of heroes, metaphors, stories, myths, ritual, 
and ceremonies. Political structures are dominated by 
power, resources, competition and related networks. 
Human resource lead organizations focus on people and 
human nature in relation to the organization. Finally, 
structural organizations rely on goals, outcomes, tasks, 
and functions. It’s possible to see how these different 
frameworks can shape or reshape research collaborations 
operating in different settings. 


When I look back on the various researcher and 
practitioner relationships that I have built over the last 
decade or more, I can easily recall what a difference it 


makes on a research project when it is possible to build trust between myself, my research team 
and the partner organizations. I’ve learned that trust is slowly gained and even more easily lost. 
I can see how the organizational culture supported or constrained our work on different projects. 
For example, I have worked in large school districts that are so centralized and structural that 
even obtaining permission to conduct research has become a months-long process. Of course, 
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sometimes these things are a matter of structure, but other times they really are a matter of trust. 
Do practitioners “trust” the research and researchers enough to give access formally and 
informally for a successful project? Authentic collaborations are key. 


When working with large organizations and/or in large research teams, trust can be even more 
difficult to obtain. If trust is about relationships, the more relationships there are to manage, the 
more difficult it is to maintain clear communication. In this essay, I reflect on (1) how 
researchers can work with practitioners to develop meaningful research designs and (2) how 
researchers and practitioners can collaborate around the technical aspects of research. The 
examples that I use are based on experiences in qualitative and mixed methods studies. As a 
qualitative researcher, I cannot offer informed insight on how singularly quantitative studies 
might be negotiated. In this essay, I draw from my own experience, as well as from course 
discussions and other experiences in policy and practice audiences. It’s also important to note 
that I worked as a student affairs practitioner for a number of years before becoming a 
researcher, so the challenges of being a practitioner were once a constant part of my own reality. 
I continue to teach in a program that trains student affairs, advising, and outreach professionals, 
so I continue to hear about and think through practitioner realities. My research agenda focuses 
on college access and financial aid for low-income students and students of color, so most of my 
work is built in collaboration with high schools, colleges and universities, and non-profit 
organizations that provide direct service to these student populations. 


Developing Meaningful Research Designs 


A research design that is uninformed by literature is considered to be weak. Why shouldn’t the 
same thing also be said about designs in educational research that are not distinctly practical in 
context or grounded in practice? Some of the most creative and effective research designs that I 
have been a part of are the result of a process that can be described as follows. First, researchers 
develop a set of research questions and a design to answer those questions. This typically 
includes a number of tools for gathering data, which supported by previous research and a 
conceptual or theoretical framework. Then, if the research team does not already have a pre- 
selected site or set of sites in mind, they might begin to think about with whom and how it might 
be possible to develop a partnership to answer these questions and support this research design. 
A reasonable next step is that the researcher and the research site come together to make sure 
that the design is a good fit. There are usually adjustments to the design, in an attempt to further 
maximize on the researchable opportunities within the collaboration. Ideally, the researchers and 
the practitioner collaborators agree on the design, so much so that a true collaboration is formed. 


Of course, the steps that I have described above are idealistic. These prescribed steps assume 
that: 


e the researchers are willing to take the time to engage in a thorough understanding of 
the context of the research organization; 


e the practitioner-based organization is truly open to and can accommodate a research 
collaboration; 
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e the key decision maker from each organization will make sure that both researchers 
and practitioners are around the table, and finally, 


e when decisions are made, they will be implemented in a way that honors the 
knowledge and expertise of everyone. 


This last assumption might be the most important one of them all, because each of these steps 
assumes a level of trust either has been or will be established. I often think back to a 
conversation that I had with a practitioner about a set of focus groups that we had conducted with 
students and parents. We were collaborating on a project as part of a college preparation 
program that served at-risk youth in a low-income high school. My research team was the 3" 
research team to examine this program in less than five years. Their previous experiences with 
focus groups had soured them to the idea. The practitioners on site had been unfairly burdened 
to assist with participant selection and other research-related details in the past. But because my 
colleagues and I had attended to our focus groups with attention to detail and made sure that we 
recruited our own participants, and kept the practitioner team up to date on our progress along 
the way, we had changed the mind of this individual. She shared that the experience was 
“owned by your team, I was able to sit back, watch them happen, and enjoy the idea of seeing 
my students and parent participate.” The individual was excited to see our data and welcomed us 
to continue to come to campus over the next two years of the project. This practitioner wanted a 
more hands-off approach to participating in the research project, but still wanted to make sure 
that her service population was excited but also protected throughout the research project. As 
one of her co-workers shared, “These kids have enough asked of them already, and so we do we 
as program staff; we need to protect our time and their [the students] paths.” 


Trust Means Risk 


What does it mean to take risks in these organizations, especially in ones where the primary 
clients themselves might be labeled as “at-risk”? The conversation that I mentioned above 
exemplifies the awareness that I must have with working within the particular research area that I 
have chosen to pursue. Though arguably, any population could be at some level of risk when 
they share their opinions as part of research that seeks to evaluate oneself and could lead to 
change. Given this delicate web of risk, how can we include practitioners and their service 
populations as equal partners in our work? 


As noted in this section, one way is to lay the groundwork for a solid collaboration is giving the 
practitioner-collaborators opportunities to actively engage in the discussion and planning of a 
research design. I’ve outlined some of the overarching steps and given an example of 
implementation. I conclude this section with a short list of recommendations: 


1. Be prepared to explain methodological approaches and tools, as well as confidentiality 
and duty to report. These issues will come up in the discussion on technical 
collaborations, but these key items should be iterated throughout the research process as a 
means of informing the research site, but also as part of honoring the population that you 
will study. 
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2. Initiate conversations with the practitioners are who are closest to being “on ground” with 
the population that you will study. If you are studying high school students, try to meet 
with teachers and counselors and/or observe them at work. If possible, ask them how 
they might engage in answering the questions that you are studying. 


3. Meet with administrators who work with your study population. They will offer 
connected, but more distal perceptions about what should be studied and possibly what 
data-gathering approaches might or might not work at this site and the rationale for those 
recommendations. 


4. There may be cases in which you will be at a site with active and or emerging 
researchers. Given that we are in the field of education, it’s not unlikely that one or more 
individuals on site are seeking a master’s or doctoral degree. Again, it’s not likely that 
they will be leading the work, but they are likely to want to have more engaged 
conversations about the design. 


Either way, it’s crucial to ask for collaboration. You might not receive it, but at least you will 
have given the organization a chance to authentically participate. At some points in writing this 
essay, I’m feeling a little preachy, and I don’t like the way that this feels, but it seems like it’s 
worth it to lay out these basic recommendations. I once worked at a high school site studying 
college preparation and college access for over a year. I met with the guidance counselors at the 
beginning of the year and had in-depth discussions about my research questions and research 
design. They had little feedback for the research design except to say that I need to be aware of 
the nuances of their student population. This was valuable advice and I made sure to use it. I 
sent updates about the work throughout the next nine months on that campus. I never received 
acknowledgement or a response. It wasn’t until my 2" to last email when I asked to meet to 
share results that I received a response from the guidance team to share my preliminary findings. 
During that meeting, a counselor shared that he had stopped reading my emails closely after the 
first few because he “knew the research plan and didn’t have time to check in.” He sort of 
apologized, but again because he knew the plan, he felt informed enough to trust me for a long 
stretch of the project without checking in. In the next section, I turn to a discussion of how 
researchers and practitioners can work together on the technical aspects of research. 


Collaborating on the Technical Aspects of Research 


I once worked on a short-term project with a fellow researcher who was about two years ahead 
of me in the field and thought that because of that I should be considered “junior” to her. She 
consistently ignored my ideas and would give me the “grunt work” administrative tasks to lead 
while she headed the more creative and analytical processes. Now, why I allowed her to do that 
is one issue, but the end goal for me was that I did not want to work with her again. I’ve seen 
these kinds of things happen between practitioners and researchers—it’s not a good thing. I tell 
this story here because in this section, when I write about the “technical” aspects of research, I 
want to be clear that this part of a collaborative relationship does not mean that practitioners 
order the coffee and cookies, while the researcher comes into interview the program 
administration. There are layers of problems with these kinds of uneven collaborations, but I 
want to highlight a few examples of technical collaborations that have been successful. 
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I am currently on a project that examines a state-wide set of learning communities. Our 
qualitative research team meets regularly with the program directors at each site. The research 
team is able to respond to any questions that the program directors might have. We are able to 
gather advice from them and ask clarifying questions about upcoming site visits. The research 
team consistently uses their advice to inform the next steps in our data collection. We may 
observe additional events or review additional documents because of their valuable expertise. 
Their advice truly guides our work. 


In another recent project, I worked with a non-profit organization on a short term, small-scale 
evaluation of their scholarship program that was administered through multiple sites. The 
protocol for the evaluation was developed in collaboration with the non-profit organization and 
pilot tested, but I still felt like something fell short within the protocols and that I might be 
missing a key piece of the puzzle. Because this was a short term, small-scale project, there was 
not a lot of time to do additional work in preparation for my interviews with the scholarship 
program administrators. During the interviews, one of the most powerful questions from the 
interview protocol was “is there anything about the [scholarship] program that I didn’t ask you, 
but you think I should know more about?” This one question empowered the administrators to 
share a wealth of information about their concerns, hopes, and ideas about the value and possible 
improvements to increase the purchasing power of the scholarship awards. 


The examples noted here may seem like small adjustments or actions, but they give voice and 
power to the practitioners. Some additional considerations are noted below: 


1. Ask for advice and information, but don’t assign tasks. When working in research 
settings, it’s likely that the individuals that you are collaborating with already have full 
work commitments. Expecting them to do additional work is not a necessary element of 
collaboration. 


2. Don’t expect that your practitioner collaborators have research experience. They may 
have important thoughts on the context of an organization, which can lend itself to the 
design, they may also understand how certain aspects of research works, but that doesn’t 
mean that they will understand the technical aspects of research, nor should they be 
expected to assist you in conducting research. In other words, gathering data or 
announcing an interview opportunity is ok, expecting them to recruit interviewees is not. 


3. At the same time, if there are individuals at a site who are very interested in learning 
more about research and its implementation, seek to find spaces for them to appropriately 
participate. 


4. Finally, part of being collaborative, maybe especially in the technical aspects of research, 
means communicating your work, including the process of your work as it occurs. As 
mentioned in the example above, it may be as simple as asking a practitioner on a regular 
basis—what should I be doing that I haven’t yet thought about? 


Collaborations involving the technical aspects of the research can occur in large and small ways. 
When practitioners and researchers are open to ongoing dialogue about flow and process of data 
collection, new ways of investigating a problem and deeper levels of understanding about a 
project may emerge. An ongoing respectful request for collaboration is one of the most 
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meaningful ways to continue to maintain trust and honor their commitments to the research as 
well as your own. 


Final Thoughts 


This piece was difficult to write, because it required a level of vulnerability and self-reflection 
that can sometimes be risky as a researcher. Researchers in the academy are socialized to be 
entrepreneurial and focus on making the most strategic of relationships. In that way, developing 
practitioner-based collaborations and really focusing on their development might mean that one 
would have to “sacrifice” more traditional accomplishments as a result of engaging in the messy 
edges of a collaborative project. I will always appreciate my colleagues in research and in 
practice arenas who are willing to make these choices in hopes of answering the often messy 
problems present in collaborative action-oriented research. 
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COMMON GROUND AND UPWARD BOUND: 
LESSONS FROM A CROSS-INSTITUTIONAL 


COLLABORATION 


BY EZEKIEL KIMBALL, TYSON ROSE, YEDALIS RUIZ, AND RYAN WELLS 


Introduction 


Federal TRIO programs provide vitally necessary 
educational programming designed to increase access and 
equity for historically underrepresented and underserved 
populations. By design, they also employ innovative Research findings demonstrate 
strategies grounded in best practices and rigorous the importance of shared 
empirical research. However, TRIO programs also pose a 
paradox: while they present potentially fruitful sites for 
research, seemingly very little is known about their impact 
and what is “known” may be confusing or even incorrect 
(Cahalan & Goodwin, 2014). The framing for the recent 
joint undertaking between ASHE and the Pell Institute 
suggests one reason why: collaborations between higher 
education researchers and TRIO practitioners are 
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understandings, or “cognitive 
common ground” in 
collaborations. Yet, college- 
going is complex, and 
potential differences in 
perspective among scholars, 
practitioners, and students 


may easily arise. In this 


relatively infrequent. 


In this essay, we highlight two related potential reasons for 
the infrequency of collaborations in research and practice: 
1) the divergent needs and interests of scholars and 


paper, we use insights from 
practice and relevant research 
to. describe how common 
ground can be nurtured. 


practitioners; and 2) a difficulty in creating the sort of 
shared meanings that would allow collaboration to occur. 
We then describe the development of an ongoing 
collaboration between personnel in the higher education 
program and the Upward Bound program at the University of Massachusetts Amherst, 
highlighting the ongoing conversations and reconceptualizations of research design necessary to 
make this collaboration work. We describe the collaboration from both perspectives—scholar 
and practitioner—affording an opportunity to highlight the initial differences in motivation for 
participation and ultimately the similarities in values for the work. 


Problems in Scholarship and Practice 


Messages received from professional conferences, literature on best practices, and graduate 
training exhort those working in higher education to think of themselves as scholar-practitioners 
(e.g., Bensimon, 2007; Love, 2012; Reason & Kimball, 2012). This knowledge base reminds 
interested parties that evidence-based decision-making is facilitated by deep engagement with 
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relevant theory and empirical evidence. However, it can also be enhanced through practical 
wisdom gained over a professional lifetime of experience. Evidence suggests that this sort of 
deep, reciprocal engagement between scholarship and practice results in improved student 
experiences; however, an even larger literature base demonstrates how very difficult it is to blend 
effectively the pressures that motivate scholarship and practice (e.g., Bensimon, 2007; Bloland, 
Stamatakos, & Rogers, 1994; Kezar, 2000; Love, 2012). 


To address this issue, higher education scholars have proposed a number of theory-to-practice 
models—including guiding concept approaches (e.g. Love, 2012; McEwen, 2003), process 
models (e.g. Evans, 1987; Stage & Dannels, 2000), and integrative models (e.g. Kimball & 
Ryder, 2014; Reason & Kimball, 2012;). Much of this literature treats the need for theory-to- 
practice conversions as a problem confronted by individual practitioners. This framing treats 
theory-to-practice conversions as an extension of the innate human ability to make meaning of 
new situations, take action, and learn from the outcomes of one’s actions. Less frequently, 
theory-to-practice models acknowledge the importance of team-based work, but still make the 
assumption that the group is comprised of professionals who share similar value commitments 
and goals for whatever project is being discussed. However, that assumption is naive and 
fundamentally problematic on several grounds. 


Teams addressing complex problems in higher education of the sorts confronted by TRIO 
programs are likely to bring together complex coalitions of students, faculty, program staff, 
administrators, and community members. The role construction of these disparate constituencies 
differs markedly (Kerr, 1963 / 2001). For example, students likely will be most interested in 
issues that directly impact them. Faculty members may be most interested in academic matters or 
could be disengaged from conversations about TRIO programs entirely. Further, while staff 
likely will look after the quality of the program as a whole, administrators have a broader 
institutional view that may conflict with the immediate best interests of TRIO programs. Finally, 
community members may seek to use colleges and universities to pursue broader economic or 
community engagement opportunities. These different roles produce markedly different views 
of the organization that, when coupled with a person’s previously held values and prior 
experiences, means that they may understand the purpose of TRIO programs differently from 
one another. For example, a belief in the importance of social justice might be shared among 
TRIO collaborators, but the way that they operationalize these ideas is likely to differ based on 
differing roles and personal experiences. As a result, intentional conversations are necessary to 
surface these differences and the assumptions upon which they are based (Bensimon, 2007). 


Searching for Cognitive Common Ground 


As described above, solving complex problems in higher education requires the integration of 
multiple perspectives and forms of knowledge. Such undertakings are fundamentally similar to 
the research process for scholars bridging multiple disciplines—particularly those who engage in 
the sort of research that focuses on pressing social problems (Moran, 2010). Consequently, in 
this section, we describe literature related to “grounding”—a theoretical construct developed by 
cognitive psychologists to describe effective communication strategies. Prior research has 
employed this approach to study how interdisciplinary collaboration works and to explain 
radically different productivity levels of research teams (e.g. Bromme, 2000; Repko, 2007). 
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As described by Clark and Brennan (1991), grounding involves the cultivation of “mutual 
knowledge, mutual beliefs, and mutual assumptions” (p. 127). That “shared information” then 
becomes the basis for conceptual common ground. However, as Clark and Brennan (1991) also 
note, the shared information required for the existence of common ground changes moment-to- 
moment. Consequently, all communicative exchanges involve ongoing updates to the basis for 
common ground. The fact that common ground is constantly evolving makes it possible for 
human beings to communicate effectively across a wide range of differences but also makes it 
difficult to comprehend when a lack of shared information exists in the moment. Essentially, 
human beings are neurologically hardwired both to create common ground and to assume its 
existence (Clark, 1996). However, while grounding has its basis in individual physiological 
traits and imperatives, it is also a fundamentally social process—the rules to which are learned 
early in life and reinforced through routine human interactions (Bangerter & Clark, 2003; Clark 
& Wilkes-Gibbs, 1986; Davies & Katsos, 2010). 


Perhaps not surprisingly, the absence of common ground in communication has been associated 
with a variety of deleterious outcomes. Beers, Boshuizen, Kirschner, and Gijselaers (2006) 
describe the inherent problems of bringing together professionals of diverse backgrounds to 
address problems of practical significance as one of “multiple ignorances” (p. 532). A person’s 
idiosyncratic and perspectival viewpoint can prevent them from understanding the equally 
idiosyncratic and perspectival viewpoint of others (Bromme, Rambow, & Nuckles, 2001). 


The importance of shared information has led many to adopt one form or another of the 
“common ground technique,” which in its most essential form is a way of creating and 
acknowledging jointly-held beliefs (Clark, 1996; Repko, 2007). As made clear above, however, 
the common ground technique merely formalizes a process in which human beings routinely 
engage, but which may not be fully completed in the face of complex problems (Beers et al., 
2006). The common ground technique simply involves: 1) the structured sharing of information; 
2) opportunities for clarification of meaning; 3) the negotiation of joint meaning, and 4) an 
agreement to revisit mutually-agreed upon common ground on an ongoing basis. 


As it applies to collaborations among researchers, program staff, and TRIO participants, 
literature on grounding helps one to understand better the potential for miscommunication and 
the need for proactive work to ensure the creation of shared meaning. Key strategies to facilitate 
this process include: 1) regular, ongoing conversations outside of the context of pressing issues 
associated with programs or research projects; 2) the mutual development and refinement of 
shared principles, goals, and strategies for action; 3) a shared willingness to revisit and revise 
these principles, goals, and strategies whenever the need arises; 4) a tacit assumption that when 
problems arise they do not stem from malevolent intent but rather from an ignorance of full 
impact on others (or more likely “multiple ignorances’’), and 5) an intentional orientation to these 
shared principles as new parties are introduced the relationship. 


Building a Collaborative Partnership 


Our collaboration began in spring 2014. The initial idea for the partnership originated with Zeke 
and Yedalis while brainstorming possible research sites for an exploratory study on college 
access. After discussing the need for research that could enhance understanding of how to 
rectify structural inequities in the college-going experiences of students from underrepresented 
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communities, they agreed that the most effective research design would involve collaboration 
with an existing access program that utilized critical praxis to both challenge and help students 
navigate institutional barriers to college attendance. 


Both team members had prior experience doing action research (Stringer, 2007) and were 
interested in producing scholarship that informed practice (Reason & Kimball, 2012). Having 
previously worked with Tyson, the University of Massachusetts Amherst’s Upward Bound 
program director, Yedalis suggested that the Upward Bound program could be a good fit. They 
reached out to Tyson to share initial ideas to develop a collaborative research program that 
would examine the influence that program administrators, high school students, high school 
personnel, college tutor mentors, and university administrators all had on the college-going 
process. Given their action research backgrounds, they also hoped to do this research in a way 
that directly benefited the program. 


While Yedalis and Zeke knew that collaborative research could pose a burden to programs, they 
were confident that the benefits of their proposed study would far outweigh the negative 
consequences. At the outset, however, Tyson had no way of knowing any of this information. In 
retrospect, it is perhaps unsurprising that early conversations proceeded in fits and starts as all 
involved began to define the scope of the project and to implicitly (but not yet explicitly) take 
part in mutual development and refinement of shared principles, goals, and strategies for action. 
As has been noted elsewhere (Clark, 2008), many community-based organizations and programs 
get fatigued by academic inquiries and research that do not lead to an actual benefit for the 
community being served. At these early meetings, Tyson expressed a desire to protect the best 
interests of the student program participants and also questioned the potential benefits for the 
Upward Bound program. Importantly, he also asked for time to consider how this potential 
partnership could best include the program’s needs before allowing access to the program and to 
be involved in defining the scope of the proposed research. 


Working with Tyson, Yedalis and Zeke prepared a series of project summaries that described 
study goals, theoretical frameworks, and research designs. These documents helped to refine the 
study’s overall purpose and research questions, and even more importantly, the process of 
drafting them resulted in a collaborative working relationship and shared understanding of the 
research project. After several weeks, Tyson, Yedalis, and Zeke all agreed to a draft project 
summary. Once this draft summary was complete, Yedalis and Zeke attended an Upward Bound 
program meeting where they met the rest of the full-time staff members. Just as the introduction 
of Tyson’s thinking on the project reshaped it in important ways, the conversation with the 
program staff did as well. This meeting led to ongoing conversations about program well-being, 
how researchers would impact interactions with students, and whether this work would be 
helpful with evaluation activities. It also resulted in a number of modifications to the project 
summary and a collaborative working relationship with the members of Upward Bound 
program’s staff that meant that they were comfortable providing ongoing feedback about the 
research project and its impact on program operations. 


Ultimately, the process from idea-to-agreement took almost a full semester, but it resulted in a 
study design that produced higher quality findings that were grounded in the real experiences of 
program staff and participants. The ongoing collaboration and development of shared meaning- 
making proved a critical component of its success. Through the process, it became clear that the 
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project’s common ground would be a shared focus on the students and their success. It also 
became clear that those approaching this commitment from both a research and a practice 
perspective were all interested in better understanding the mechanisms by which participants in 
the access program form college-going aspirations. 


By design, the project aimed at generating actionable findings that could inform policy, 
budgetary, and programmatic decisions. A secondary aim was to gain knowledge about the way 
that students form, reshape, and nurture a college-going identity in relation to structural obstacles 
and personal experiences. Additionally, the program leaders identified a need for investigating 
key components associated with student participation in the access program specifically related 
to student participant satisfaction and elements of social support. The project approach included 
multiple methods for investigating these key components including interviews, focus groups, 
observations of program activities and document analysis. 


The communication established in the initial stages of the project continued as the research 
began in earnest, and was instrumental in the success of this work. Yedalis and another graduate 
student researcher regularly attended Upward Bound program meetings and spent a great deal of 
time as observers at the program site. When the research team was on site at the access program, 
in addition to the interviewing or field observations, they would join in program activities and 
operations as needed, which made them useful to the program staff in multiple ways. This work 
helped them not just to develop better rapport with the Upward Bound program staff but also a 
better understanding of the Upward Bound program—and how it differed from others around the 
country. 


By working closely in an integrated fashion with the program, it became clear what additional 
resources would be useful to the program and offered an opportunity for developing access to 
other community networks and services. For example, the students in the program identified an 
interest in interning in the field of public health directly connected to food access and healthy 
behaviors. As a result, Yedalis was able to make a connection to a community health center with 
which she had a long-standing professional connection and knowledge of their regular work with 
community partners. The health center and the Upward Bound program met to explore future 
health programming collaboration, including possible opportunities for student internship 
programs. Although the health center and the high school where the Upward Bound program is 
based are within walking distance of each other, it was through this collaborative research 
project that they became aware of common ground in their respective programming and goals. 
This example demonstrates the opportunities for developing new members within a collaborative 
project and identifying additional resources and opportunities for meeting research and 
programmatic aims when all involved have an “intentional orientation” to the shared principles 
of common ground discussed above. 


Addressing Concerns Regarding Impact on Practice 


Throughout this collaboration, those involved returned frequently to the idea that community- 
based action research should be more focused on the community served than on the research 
results. The idea served as the common ground that held our collaboration together (Bromme, 
2000). As a result of this shared understanding, people with disparate motivations and interests 
could work together on a research process that resulted in outcomes meaningful to all involved. 
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This collaboration is consistent with Wenger’s (1998) community of practice, which is a 
collection of individuals and groups sharing a common purpose and learning together in the 
service of that purpose. Participants in a community of practice share their knowledge, 
experiences, and resources to achieve meaningful goals. 


In our collaboration, common ground occurred more easily due to the prior backgrounds and 
experiences of the personnel involved. As Clark and Brennan have noted (1991), common 
ground has to be built from shared systems of meaning. Both Yedalis and Zeke identify as 
scholar-practitioners rather than researchers and value community-based work. When the 
Upward Bound staff sought definition and redefinition of the work Yedalis and Zeke proposed to 
do, they saw it as an opportunity to revisit and revise the principles, goals, and strategies the 
partnership was based upon rather than an imposition. Doing so is consistent with Bensimon’s 
(2007) admonition that examining the implicit theories utilized to structure practice is critical to 
cultivating student success outcomes. Further, Yedalis’ extensive professional experience 
working in community-based education, and her personal experiences as a bilingual/bicultural 
first-generation student participating in programs such as this one made her uniquely well-suited 
to engage in this sort of research. Therefore, her ability to meet and join the access program staff 
in a conversation about a potential collaboration included elements of membership that helped to 
bridge the research and programmatic aims. 


However, even given the researchers’ backgrounds and commitments, the collaboration would 
not have been possible without the ongoing commitment, expertise, and willingness to find 
common ground of the Upward Bound program staff. As with many grounding processes, this 
commitment took considerable time (Bromme et al., 2001). Here again, the prior experiences of 
the personnel involved proved helpful. Tyson readily identifies as a scholar-practitioner and is a 
graduate of an academic program closely related to the one that serves as Yedalis and Zeke’s 
academic home. Further, he is thoughtful and intentional in his approach to programming: 
virtually all of the work that he does is informed by critical and social justice perspectives. 
These commitments were infused into the programmatic and staffing decisions. Consequently, 
while deeply committed to the best interests of the program, Tyson and the Upward Bound staff 
were already positively predisposed to a project of this sort—provided it could be framed in the 
right way. Perhaps more importantly, they had the “intentional orientation” needed to engage 
with the process, and with others with different roles, experiences, values, and assumptions about 
the work. This observation is consistent with literature that suggests that the creation of common 
ground is a complex, negotiated process (Beers et al., 2006). Both the research team and the 
Upward Bound program staff members also worked to make students active participants in the 
research by sharing with them the purpose of the work and allowing them to help shape research 
in important ways. For example, student participants first suggested the importance of fear as a 
motivation for both college-going and dropping out. They also helped to shape the research 
methods on this theme in important ways—for example, providing suggestions for new research 
topics and letting researchers know when they felt they had something to contribute via an 
interview. 


Importantly, Tyson and Yedalis’ prior working relationship effectively established a baseline 
level of trust and provided an opening for conversation. This statement is consistent with 
literature on common ground that describes it as a dialogic process (Bangerter, & Clark, 2003). 
This conversation proved to be critical in addressing Tyson’s well-founded skepticism. The 
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Upward Bound program staff members had built a program infused with a commitment to social 
justice—committed to recognizing and working to address the effects structural racism, poverty, 
and other forms of oppression have on the daily life of program participants. Consequently, the 
Upward Bound program staff members are very intentional in looking at practices, 
collaborations, relationships and processes that stem from and embody shared philosophical and 
ideological beliefs. As noted above, prior literature has shown that structured thinking about 
program purposes and design produces positive outcomes (Reason & Kimball, 2012). Before 
approving Yedalis and Zeke’s proposed research, Tyson and the program staff needed to develop 
a deeper understanding of the perspectives of the research team. This process helped to address 
concerns regarding the program staff's fear that the research team would not value the expertise 
of the practitioners or program participants, the prospect that research would only serve the 
academic designs of the researchers, and the perception that the research process would be 
intrusive. In so doing, concerns that academic research might not prove useful were overcome 
(Kezar, 2000; Love, 2012). These concerns are quite real for TRIO programs—given a paucity 
of available empirical information. 


Ultimately, Tyson and the Upward Bound program staff grew comfortable with the researchers 
and the design that they proposed through sustained dialogue. However, that comfort alone 
would not have been sufficient motivation for them to allow the researchers access. The 
program staff members were also concerned about potential adverse impact on student 
participants. These concerns stem from the recognition that many researchers may not possess 
necessary cultural competencies to work with students from marginalized communities (see 
Bensimon, 2007). To address this concern, the design proposed by the researchers needed to 
address pressing issues of program concern while also taking into account the knowledge, 
experiences, and backgrounds of program participants. Based on a design that did so, the 
program staff gained access to information and research capacity to which they would not have 
otherwise had access. 


For under-resourced TRIO programs, these sorts of collaborations can be particularly fruitful. In 
this case, the program has been able to investigate the experiential and anecdotal knowledge of 
practitioners and provide evidence that is necessary to implement programmatic interventions 
and positively impact student outcomes. The research also offered the opportunity to delve 
deeper into and understand the student experience without expending valuable internal resources. 
Additionally, the research has offered the program the critical inquiry and feedback 
opportunities. What emerged through the intentional process of developing, deepening and 
maintaining common ground was a research collaboration that represented the rigor and 
mindfulness necessary to recognize and attend to the concerns held by the program staff while 
also meeting researcher needs. It created the opportunity whereby the researchers and the 
program gained valuable insight and knowledge that will impact the program across multiple 
levels of the organization. Ultimately, what was created was an environment where the 
reciprocal goals of research and practice could be realized. 


Conclusion 


Reciprocal engagement between scholarship and practice offers opportunities to share scarce 
resources and vital insights. However, to successfully collaborate with people who have 
radically different role constructions—scholars, practitioners, students and community 
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members— requires intentional relationship building in order to establish lasting partnerships. 
First, the divergent assumptions, values, and priorities of those involved must be acknowledged 
and respected. Second, common ground in purpose must be established, explicitly explored, and 
systematically revisited. Third, a shared group identity that is simultaneously enduring in its 
general purpose and flexible in its members must be created based on mutual trust. Finally, the 
collaboration must foreground actual impact—what works and what does not. None of this work 
is easy, but it is possible. 


The collaborative project that we described in this essay achieved common ground, which has 
helped it to continue despite conflicting demands and priorities. Working to establish a 
community of practice engaged in a single community-based research project has yielded 
broader dividends. Through ongoing discussions regarding resources and common purpose, the 
original goals of the conversation have shifted to the potential for larger impact. As a result, a 
subsequent collaborative was developed through this process—the Access Pathways Project 
(APP), a collaboration between the Upward Bound program and key departments within the 
university that are committed to higher education access among underserved populations. It also 
provided an opportunity to bring in other researchers—Ryan Wells among them—as we sought 
to understand student success not just in accessing higher education institutions but in flourishing 
within them once enrolled. This project illustrates an example of the process of developing a 
collaborative partnership and describes the key factors to establishing a mutually beneficial, 
trusted and sustainable partnership between researchers, practitioners, and community. 
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RESEARCHER AND PRACTITIONER COLLABORATION 


TO ADVANCE INQUIRY AND ITS USAGE sy aneeta seu, 


ROBERT ANDERSON, GEORGIA HUGHES-WEBB, AND ADAM S. GREEN 


Abstract 


This essay explores the 
relationship between 
researchers and_ staff of 
college access and success 
Although 
collaborations between the 


programs. 


two are at times 
challenging, the outcomes 
are significant for all. The 
essay goes on to show a 
case study example from 


Introduction 


Researchers and evaluators of college access and success 
programs come to their work armed with knowledge of both 
research methodologies and the field gained from their 
education, experience, and immersion in literature and 
research. As they begin framing a study, gathering 
background information, and planning data collection, 
however, researchers are faced with an aspect of their work 
for which schooling and research literature provide little 
guidance—working with program staff. Researchers are 
dependent on and must work with practitioners of college 
access and success programs to gain crucial information 
ranging from contextual information to student participation 
and outcome data. This essay argues for a collaborative 
approach to conducting research on these programs both for 


the values collaboration embodies and the benefits it offers. 
While it may be more applicable to qualitative inquiry and 
program evaluation, these are solid precepts for quantitative 
inquiry as well. We explore why relationships between 
researchers and staff of college access and success programs 
ss can be difficult but argue that the benefits for all are 
————— significant. We follow with a brief case study of the West 
—_—_—_— Virginia GEAR UP program evaluation and conclude with 


recommendations to be applied to research more broadly. 


the West Virginia GEAR 
UP, and concludes with a 
list of recommendations. 


Challenges to Collaboration 


Educational researchers, practitioners, and other stakeholders are invested in understanding what 
works in college access and success programs. So why is the relationship fraught with 
challenges? While their area of work is similar, the choices made in how to engage in that work 
are different. Choosing to be a researcher versus a practitioner belies different orientations and 
even different values that can lead to differences of opinion and priorities. Both groups possess 
knowledge and experience about these programs from their education and work. This deep 
knowledge can work to prevent one from seeing the different knowledge and perspectives of the 
other and how the two might be complementary. Research ethics themselves may deter 
researchers from developing relationships with program stakeholders and staff. Researchers may 
fear for real or perceived lack of objectivity due to relationships with the people implementing 


60 


the program of study. Another substantial challenge to building relationships between 
researchers and practitioners is simply time. Jointly framing inquiry, understanding perspectives, 
and collaborating take a significant amount of effort and time that must be prioritized in order to 
be coordinated and carried out. From the perspective of the practitioner, being an active 
participant in the research process can be seen as time taken away from actually implementing 
the program. 


Perhaps the biggest obstacle in fostering relationships and collaboration between researchers and 
college access and success program staff is the development of trust. Practitioners may be 
skeptical that research can adequately capture the program’s complexity and the constraints and 
context that frame its outcomes. Practitioners have responsibility for program success and have 
loyalty to its stakeholders, employees, and participants. This can foster very real fears about the 
uses of research and how it will make the program look, especially to funders. Unfortunately, 
such fears could lead to curtailed participation and candor. Researchers then may fear they are 
not getting complete and unbiased information. 


Consequences 


When mistrust prevents good relationships, research or evaluation is something done to rather 
than with the program, resulting in less than full cooperation and incomplete information and 
data provision. This threatens the validity of the research and limits the likelihood of appropriate 
conclusions and recommendations. A poor relationship and a sense among practitioners that 
they have been the subject of assessment as opposed to participants in a learning process 
decrease the chances of recommendations actually being implemented. 


On the other hand, a good relationship and collaboration lead to better research design, more 
complete information and data, and therefore higher quality research. Ownership of and 
participation in research processes by practitioners increases the acceptance of findings and 
conclusions and the application of recommendations (Patton, 1997). A unified front between 
researchers and practitioners about recommendations can also enhance the response from 
decision makers, funders, and policymakers. On a more philosophical level, working 
collaboratively shares power and responsibility for research design and decisions between 
researchers and program practitioners (Patton, 2002). This puts into practice the sense of equity 
that drives educational access and success researchers and practitioners in the first place. More 
globally, it holds in balance professional principles of competent and systematic inquiry with 
respect for people both immediately and indirectly affected by the process and outcomes of 
research (American Evaluation Association, 2004). 


While we might philosophically agree that collaborative research and evaluation are good and 
have clear benefits, actual implementation requires planning, communication, flexibility, and 
responsiveness to feedback. Following is a case study of how the staff and evaluators of a 
college access program collaborated to improve both the evaluation and the program itself. 


Case Study 


Background of West Virginia GEAR UP. In July 2008, the West Virginia Higher Education 
Policy Commission (WV HEPC) was awarded a U.S. Department of Education GEAR UP grant 
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totaling nearly $18 million. GEAR UP (Gaining Early Awareness and Readiness for 
Undergraduate Programs) aims to help more students prepare to be successful in some form of 
education and training beyond high school. 


The 2008-14 WV GEAR UP grant was comprised of two structural components: (1) annual early 
academic interventions for a cohort of 7" grade students who would graduate high school in 
2014 (called the 2014 Cohort) and (2) a priority student approach focusing on just-in-time 
outreach interventions for 11" and 12" grade students in need (called the Priority Group). Over 
the six-year period, more than 13,000 unduplicated students received services in a total of 48 
public schools. College access services included, among other interventions, tutoring, 
mentoring, college and job site visits, financial aid advising, and summer residential academies. 
Parents and family members participated in college and career planning, including FAFSA 
completion workshops. Educators in GEAR UP schools participated in meaningful professional 
development aimed at helping them prepare students to aspire for and be academically prepared 
for postsecondary education. 


Proposed WV GEAR UP Evaluation. Like many college access and success programs, the 
WV GEAR UP project included broad goals to ground its program evaluation. Specifically, WV 
GEAR UP sought to significantly increase (1) the academic performance and rigorous 
preparation of 2014 Cohort students for participation in postsecondary education; (2) the number 
of high-poverty, at-risk students prepared to enter and succeed in postsecondary education; (3) 
the rate of high school graduation and participation of 2014 Cohort students in postsecondary 
education; and (4) 2014 Cohort students’ and families’ knowledge of postsecondary educational 
institutions, admissions requirements, and financial aid. WV GEAR UP also had a fifth goal that 
aimed to ensure effective, efficient, and appropriate use of resources through intentional 
collaboration with identified partners committed to GEAR UP goals. 


To determine the extent to which the program achieved its five broad goals, the evaluation plan 
included in WV GEAR UP’s grant proposal transcended a design aimed simply at answering 
questions required by the Federal Annual Performance Report (APR) due each April (e.g., 
number of student and parents served, perceptions of college affordability). WV GEAR UP 
proposed a rigorous evaluation that was grounded in research questions framed to address both 
project implementation (the quality and quantity of project activities and interventions) and 
impact (the degree to which project objectives and goals were achieved). According to 
preliminary plans, implementation findings were intended to allow project managers to make 
formative decisions about activities and interventions on a day-to-day basis to provide the 
greatest benefit to stakeholders. However, the ultimate evaluation plan was designed to look 
across project implementation and outcomes to make summative statements about the project as 
a whole. 


Early Challenges: Building a Trusting Relationship. WV GEAR UP selected an external 
evaluator through a Request for Proposals process based on the evaluation company’s model to 
evaluate the impact of the program through effective data collection, creation of a web-based 
data warehouse for all data sources, and reporting of program implementation and outcomes data 
beyond APR requirements. 
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Developing a trusting relationship between WV GEAR UP practitioners and the selected external 
evaluators was a challenge at first. This was partially because the program was well underway 
by the time the evaluator was awarded a contract (January 2009). The program had begun in 
July 2008, giving program staff and local stakeholders several months to prepare for 
implementation before the evaluator was brought on board. 


At the outset of any project, much coordination and planning are required to establish 
expectations and timelines to which all parties agree. In the case of the WV GEAR UP project, 
program staff had to undergo the process twice—once following the grant award and again after 
the selection of the evaluator. Although the repeated process required time and effort for 
planning and coordination at two different points in time, this coordination was important to the 
integration of the parallel activities of the grant and its evaluation. The proposed evaluation plan 
included a variety of methods and strategies for collecting and analyzing data. Following award 
of the contract, the nature and scope of the data to be collected had to be finalized, data 
collection tools had to be designed, and strategies and protocols for collecting the information 
had to be established. 


Although WV GEAR UP staff at the state level understood and appreciated the need for 
comprehensive data collection, site coordinators (local liaisons responsible for planning and 
implementing services) did not have the same understanding of the need or appreciation for the 
possibilities the information might afford them. Site coordinators also did not understand why 
survey response rates were set so high (80% for students and 50% for parents). Repeated, 
consistent explanation of the importance and need for the information helped site coordinators 
become active participants in data collection, particularly for surveys; however, some continued 
to perceive the magnitude of data collection to be a burden. 


Through their efforts, WV GEAR UP staff learned that reaching down from the higher education 
sphere into the K-12 sphere to provide and evaluate college access programs is not a successful 
model. The approach must be collaborative and inclusive, valuing the perspectives, experiences, 
and expertise of stakeholders at both levels. If all parties feel they are valuable to the effort, they 
will be more likely to value and utilize the available data to make decisions and scale best 
practices. 


Cultivating a Strong Relationship. Because there were minor bumps in the road at the 
outset of the WV GEAR UP evaluation—as in most studies of a similar scope and complexity— 
WV GEAR UP staff and the external evaluation team worked intentionally to develop a 
relationship that was mutually agreeable and capable of producing high-quality, useful 
information. Ultimately, the shared goal of implementing a strong research design and 
developing valid and usable knowledge about the program’s implementation and outcomes 
united program staff and evaluators in a common effort. The evaluator’s general philosophy of 
collaborative, utilization-focused, and pragmatic (Patton, 1997, 2002) evaluation practice likely 
gave the team a greater propensity toward cooperation with program staff than researchers who 
subscribe to other philosophies. The philosophical outlook alone, however, did not result in a 
strong relationship or a high-quality evaluation. Both the program and evaluation teams took 
explicit action to build, refine, and reinforce the partnership. 
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Communicating Regularly. An important and fruitful strategy for building understanding 
and trust between WV GEAR UP staff and evaluators was regular interaction through in-person 
meetings and other forms of communication. Evaluators often met with the WV GEAR UP 
Advisory Board to discuss evaluation progress and findings. They also attended monthly site 
coordinator meetings to provide information or materials, collect data, and answer questions. In 
addition to formal interactions, WV GEAR UP staff and evaluators interacted regularly through 
in-person office visits (both scheduled and impromptu), telephone calls, emails, and text 
messages. The frequent communication helped ensure the evaluation team had a comprehensive 
understanding of program administration, operations, and priorities at all levels of operation. 
Likewise, program staff developed an extensive understanding of evaluation processes, 
requirements, and constraints. Maintaining regular communication promoted responsiveness to 
emerging needs (e.g., programmatic or evaluative adjustments, data requests from policy 
makers). Further, the varied communication methods allowed both partners to see and hear the 
knowledge and passion each brought to the partnership. 


Over time, the interactions between WV GEAR UP staff and evaluators helped members of each 
team get to know each other personally. These personal relationships further enhanced the sense 
of trust and respect. Some researchers and evaluators may be hesitant to develop personal 
relationships with program stakeholders due to a fear of compromising their objectivity. 
However, in the case of the WV GEAR UP project, the development of personal relationships 
across teams was in keeping with traditional Appalachian cultural values (e.g., hospitality, 
personalism; see Jones & Brunner, 1994) and promoted mutual trust and appreciation for the 
knowledge and perspectives of each team. Neither team perceived that research objectivity was 
compromised through positive personal relationships. The fact that the Year 6 WV GEAR UP 
evaluation report won the 2015 AERA Division H Outstanding Publications Competition in the 
Program Evaluation category further suggests that the quality of the research was not 
detrimentally affected. 


Maintaining Flexibility. Although an evaluation plan and other documented agreements 
(e.g., contract, scope of work) formed the basis for the formal relationship, the evaluation team 
and WV GEAR UP staff acknowledged from the outset the possibility that changing 
circumstances may necessitate flexibility in preliminary plans and agreements. For program 
staff, such flexibility manifested itself in various ways: revisions to requirements of local sites’ 
annual work plans, changes in emphasis from year to year, addition of programmatic 
opportunities, and so on. Programmatic flexibility was both a necessary response to changes in 
federal requirements and a determined response to emerging needs and evidence of program 
effectiveness. One may expect that program staff will adapt or refine implementation strategies 
throughout the life of a program in order to achieve goals. However, evaluators had to remain 
mindful that programmatic changes may demand corollary adjustments to evaluation plans. 
Again, the shared goal of building knowledge about the program as implemented—complete 
with formative modifications—ensured that evaluators and WV GEAR UP staff collaborated to 
make adjustments when needed and appropriate. 


Flexibility in the evaluation was not merely a response to programmatic changes. It was also 
needed in revising the scope of work and data collection to meet WV GEAR UP’s needs. The 
integrity of the underlying research design was never compromised or altered; the quasi- 
experimental design was carried out largely as initially envisioned. However, the scope of the 
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data collected transformed over time in response to the needs of the program. Content and 
format (paper versus online) of student, parent, and school personnel surveys was revised 
carefully over time to ensure that the instruments efficiently captured all information vital to both 
the evaluation and to program staff’s evolving needs. Items in survey instruments were routinely 
reviewed collaboratively; unnecessary items were deleted, and response options were revised as 
needed. Additionally, new survey or interview items or sections were added to ensure that data 
were available for areas of critical importance to WV GEAR UP staff. In Year 3 (2010-2011), 
the annual student and parent surveys were revised to include items assessing familiarity with 
and utilization of the College Foundation of West Virginia (CFWV) web-based planning and 
information portal. Although not designed specifically for the WV GEAR UP program, the 
portal did provide information and services relevant to GEAR UP goals, and WV GEAR UP 
staff promoted its use as part of the program. Program staff needed information about students’ 
and parents’ knowledge and use of the portal to design further awareness campaigns and training 
efforts. Therefore, evaluators included relevant survey items to provide additional value in the 
evaluation with minimal additional cost. 


Changes to the scope of evaluation reporting were also negotiated in Year 1. As initially 
designed, the evaluation team was to provide quarterly summaries of evaluation activities and 
progress, along with key findings and policy implications. Comprehensive reports of findings 
and recommendations were to be delivered in Years 3 and 6. During the first year, however, the 
Chancellor of the WV HEPC expressed a desire for more regular formal reporting of 
comprehensive findings. As a result, evaluators restructured the scope of the reporting plan to 
include quarterly progress summaries three times a year, annual reports of formative findings, 
and a final summative report at the conclusion of Year 6. In revising the project’s reporting 
requirements, the evaluation team and program staff worked to ensure that changes in the time 
and cost burden was still within the scope of the original contract and budget. All of these 
instances of changes to program and evaluation activities required and exemplified close 
collaboration between staff and evaluators that resulted in taking advantage of rising 
opportunities and meeting stakeholder needs. 


Sharing Responsibilities. One of the most powerful examples of the benefits of 
collaboration between program staff and evaluators involves the administration of annual student 
and parent surveys. Each year, participating students and parents completed surveys about their 
knowledge, perceptions, attitudes, and experiences related to WV GEAR UP and students’ 
postsecondary options. Every other year, students and parents in a comparison group also 
completed surveys. Over six years of the project, 18,135 student surveys and 10,303 parent 
surveys were collected. 


The surveys provided valuable information for the evaluation; they were also a required 
component of the federal program. Federal requirements mandated an 80% response rate for 
student surveys and a 50% response rate for parent surveys. Although response rates of that 
magnitude are not entirely unattainable, they are incredibly difficult to achieve. Evaluators 
would not have been able to gain the cooperation of several thousand parents and students each 
year without significant assistance from GEAR UP site coordinators. WV GEAR UP staff and 
evaluators worked together to coordinate local survey administration through the site 
coordinators who distributed and collected parental consent forms and surveys. 
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Strategies for the consent process and survey administration changed over time both to ease the 
burden on site coordinators and to improve response rates. After the first year, WV GEAR UP 
staff and evaluators collaborated to develop a durable consent form for participants that covered 
all data needs and the entire anticipated time-frame of the research—a shift from the preliminary 
strategy of requiring a new opt-in consent form annually. As it became clear that reliance on 
paper-based surveys might pose a barrier in achieving required response rates (and posed a 
substantial burden on site coordinators), the evaluation began to incorporate online surveys. 
Different web-based survey strategies were tried (with paper surveys available as needed), and 
by Year 4, the program met or exceeded response rate requirements (see the figure below). 
Moving to an online survey format both improved response rates and facilitated easier to 
implement processes for site coordinators. These solutions met the needs of both program staff 
and evaluators. 
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Beginning in Year 3, the evaluation team provided ongoing updates about survey response rates, 
and in Years 4 through 6, evaluators provided site coordinators with real-time survey completion 
updates through an integrated survey tool and participant management system. Providing timely 
information about response rates helped site coordinators develop targeted lists of non- 
responders for follow up. Response rates steadily improved over time—and exceeded minimum 
requirements—as WV GEAR UP staff and evaluators adjusted survey administration strategies. 


Triangulating Results and Interpretations. As part of reporting processes, evaluators 
routinely shared preliminary drafts of all reports with program staff for review. Reviews 
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conducted by WV GEAR UP staff primarily served as validation checks for reported data and 
the evaluation team’s interpretations of findings. Asking program staff to review the 
interpretation of findings and proposed recommendations was an important strategy for 
evaluators to ensure they properly understood the program’s context. Some researchers may be 
hesitant to invite program staff to be part of the reporting process; they may have justifiable 
concerns about the possibility of clients asking them to modify or exclude findings. Building a 
strong foundation of trust, understanding, and respect, however, can work to prevent such 
actions. WV GEAR UP staff never asked evaluation staff to change or exclude findings. 
Program staff only reviewed findings to ensure their accuracy and completeness and to provide 
additional insight or information to assist evaluators in drawing comprehensive and contextually 
meaningful conclusions and recommendations. The strategy, built on trust, ensured that all 
appropriate perspectives were taken into account. This opportunity to provide feedback on early 
reports also fostered program staff trust in the evaluation as well as a sense of ownership that fed 
future collaboration. 


Utilization of Findings. Evaluators’ trust in program staff typically increases when staff 
make use of evaluation findings. Professional evaluators design evaluation studies to be useful 
and used by stakeholders. In fact, the program evaluation standards specifically address the 
utility of an evaluation for intended users (Yarbrough, Schulha, Hopson, & Caruthers, 2011). 
When stakeholders do not use findings of evaluation studies in meaningful ways, evaluators may 
become frustrated and skeptical about stakeholders’ motives. Throughout the WV GEAR UP 
evaluation, however, program staff regularly reviewed and used evaluation findings to guide and 
inform decision- making. At the state level, findings were used to revise annual work plans, 
which were included in annual sub-grant agreements and site coordinator contracts. At the site 
level, friendly competition for results led to incorporation of findings. For instance, at the close 
of Year 3, when results showed that for a third year, only 17% of students or fewer could 
correctly identify the cost of one year of tuition at a four-year public college or university in 
West Virginia, WV GEAR UP staff designed and implemented specific interventions to improve 
understanding of tuition costs. By the end of Year 4, approximately 34% of students and 30% of 
parents correctly identified the cost of one year of tuition at an in-state college. By the end of 
their senior year in high school, students in the 2014 Cohort were three times more likely than 
their non-GEAR UP peers to accurately estimate the cost of tuition. 


The evaluation team also used feedback to adjust not just tools and processes but also how they 
interacted with program staff. For instance, in an ill-fated attempt to bring both clarity and levity 
to a description of survey administration procedures, evaluation team members devised and 
presented a skit to demonstrate the process. Rather than finding the skit light-hearted and 
helpful, however, site coordinators found it condescending and unnecessary. Their comments on 
the evaluation form WV GEAR UP staff administered following the event clearly voiced their 
displeasure. Evaluation staff took this feedback into consideration when deciding how to 
improve interactions with site coordinators. At the next site coordinator meeting, evaluators 
specifically referenced the offensive skit (and site coordinators’ poor opinion of it) in a good- 
humored way and promised not to offer any more theatrical performances. Evaluators’ 
willingness to accept and use that feedback improved the relationship between the evaluation 
team and site coordinators, which became very positive over time. 
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Within the partnership, utilization of data and findings was only possible because both partners 
were open to critiques and suggestions. That openness was enabled and enhanced by the trusting 
relationship the partnership built throughout the life of the WV GEAR UP grant. 


Recommendations and Conclusion 


The fruitful relationship between WV GEAR UP staff and external evaluators was developed 
through intentional acts by both groups. What follows is a list of recommendations based on 
their experience. 


1. Include program staff in the development and/or finalization of research or evaluation 
plans. 


2. Evaluation/research staff and program staff should communicate regularly in varied 
formats to foster the relationship and so that each is abreast of the other’s progress and 
needs. 


3. Develop from the onset and maintain an openness to flexibility. Both program and 
evaluation/research needs require it. 


4. Share responsibilities for data collection and its improvement. Data collection that is 
less burdensome and more efficient is respectful of everyone’s time and increases the 
likelihood of comprehensive collection. 


5. Utilize formative findings. Both evaluators and program staff can improve their work 
and the relationship by doing so. 


The WV GEAR UP program and its evaluation are an example of how intentional collaboration 
can result in deeper trust between parties and a more productive evaluation relationship. The 
collaboration resulted in a high-quality evaluation that met federal requirements, responded to 
changing needs on both sides, and ultimately provided information that improved the program. 
To learn more about WV GEAR UP and its evaluation and research efforts, visit 
www.wvgearup.org 
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DESIGNING PROGRAM OUTCOMES WITH AN EYE 
TOWARDS PROGRAM IMPROVEMENT 


BY CHRISTOPHER M. MULLIN 


Years of research have identified particular issues that 
grant programs are ideally designed to address. Grant 


programs are structured to offer practitioners the SSS 
opportunity to make meaningful improvements for Abstract 
participants within a particular and often unique context 

of implementation, and funders — be they governments or This essay focuses on the way 
donors — can target funds to particular populations or programs should be 
purposes to meet their objectives in a way that general structured in order to address 
appropriations or unrestricted grant funds do not. Yet, both program improvement 
there exists an inherent tension in what would otherwise and excellence. It closes with 
be considered an ideal combination of mutual interests. a discussion about two 

In particular, I speak of the desire by practitioners to supporting factors critical to 


make improvements in program delivery and outcomes 


; ; ae the success of the process: 
and the desire of funders to achieve often very specific 


leadership and management. 


outcomes. 

How then do we structure programs to acknowledge both eo 
program improvement and excellence? The purpose of _ 

this essay is to suggest a process to address this paradox. = 


This is accomplished by first reviewing key components 

of a purposefully structured plan focused on success which includes identifying what is to be 
measured, requiring projected outcomes, comparing actual and projected data to inform program 
impact and explaining what the data mean. 


Structured for Success 


Like a building, a program cannot be sound unless it has a strong foundation. Characteristics of 
a strong foundation include a series of articulate statements that define the scope of the work, 
expected deliverables and parameters delineating allowable activities. As it relates to federally 
funded programs in particular, these characteristics are well espoused. 


A request for proposals (RFP) that includes these foundational elements allows for applicants to 
focus squarely on the activities and related expenses that will be employed to meet the grant 
objectives. Furthermore, a strong RFP allows for the development of a monitoring and 
accountability system during the proposal phase to guide the creation of proposed activities and 
align all activities across the various phases of the grant program to ensure fidelity in 
implementation. 


Vv Vv Vv wv 


Vv Vv Vv wv 


Experienced grants writers and practitioners recognize that the aligned outcomes that run 
throughout the grant process — from the RFP to project completion — are just one aspect of a 
successful application. The substantive part of any proposal and resulting reports include the 
activities undertaken by the proposed program to achieve the outcomes. Should a proposal be 
funded, successful implementation is the result of implementing the plan as proposed, 
continually monitoring its outcomes, and learning throughout implementation. 


In this section, I discuss a linear progression from grant development — building the foundation — 
through project completion with a direct focus on the use of data to ensure that the program 
focuses not only on the outcomes desired but the continual improvement practitioners seek. I 
begin with a process to identify what is to be measured and how it is measured, then continue to 
articulate the need for projections to remain focused, and close with how comparing data 
supports improvement efforts. 


Part 1: Identify What is to be Measured 


The development of outcomes measures is a process that starts with identifying goals, continues 
to the creation of outcomes that reflect the goals, which informs the creation of metrics to 
quantify the outcomes and hinges on meaningful indicators of performance. In 2015, the Post 
Collegiate Outcomes Initiative’ published a number of papers related to the post-college 


' The American Association of Community Colleges (AACC), in partnership with the Association of Public and 
Land-grant Universities (APLU) and the American Association of State Colleges and Universities (AASCU) have 
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outcomes of students. Among them was a paper examining the dimensions to consider when 
creating measures in which the terms outcomes, metrics and indicators were defined as the 
following: 


e Outcomes: the results of a higher education experience that are evaluated or measured 


e Metrics: the standards of measurement of a system of parameters used to evaluate 
outcomes, and 


e Indicators: statistics that provide a context or benchmark for metric results (American 
Association of Community Colleges [AACC], American Association of State Colleges 
and Universities [AASCU], and the Association of Public and Land Grant Universities 
[APLU], 2015, p.2) 


This essay builds upon this important work by extending the conversation around metrics to 
include both outcome and process metrics as well as re-envisioning how we think about 
indicators. This discussion is supplemented by a modified version of a table presented by the 
Initiative, shown here as Figure 2. 


collectively developed a strategic framework to guide discussion and the creation of measurement tools for reporting 
student outcomes after college. Funded by the Bill & Melinda Gates Foundation, the project partners assembled 
subject-matter experts and institutional leaders to create a framework and application tools that will enable colleges 
and universities, policymakers, and the public to better understand and talk about post collegiate outcomes in areas 
such as economic well-being, ongoing personal development, and social and civic engagement. The development of 
the framework and the accompanying tools are an important first step toward the creation of common metrics and 
indicators for use by institutions to report a more comprehensive set of post collegiate outcomes. For more 
information see http://www.aacc.nche.edu/AboutCC/Trends/pco/Pages/default.aspx 
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Outcome Outcome Metrics Process Metrics Indicators 
Program Participants Participants Weekly 
Completion completing the completing core attendance rates 
Rates program as a percent activities. 

of all participants 

starting the program. Attendance rates of 

program 

Participants participants. 

completing the 

program as a percent Percent of 

of participants in the participants 

program one month completing %, '” 

after starting. and % of the total 

program activities. 

Employment Employment in the Hours of on-the-job Weekly rates of 
Rates fourth fiscal quarter training/internship/ work completion 

after completing the externship per 

program. participant. 

Position change Acquisition of 

(promotion) for industry 

program participants certification during 

who were working at program 

time of program participation. 

enrollment. 


Outcomes. Outcomes are those results of an experience that are to be measured. Often, these 
outcomes are constructs, or concepts, that allow for laypersons to be engaged in the project 
without being over-prescriptive so as to squander creativity and alienate the support of 
laypersons or experts. Figure 2 presents two fairly common outcomes of interest: completion 
rates and employment rates. Practitioners with experience in the field fully understand that there 
are a number of ways to measure these outcomes, each accompanied by caveats and concerns. 
However, this cannot serve as a barrier to embracing the idea. What must occur to support 
meaningful grant implementation is the acceptance of a metric or series of metrics related to the 
outcome. 


Outcomes Metrics. Measurement of those actions that can be observed may be constructed in 
a way that results in some metrics, be they the counts of actions (number of participants) or rates 
(number of participants reaching a defined threshold such as completion). Outcome metrics are 
directly related to the outcome and are typically “high-level” metrics that speak to the group in 
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aggregate. Figure 2 presents an example, where outcome metrics for the Program Completion 
Rates outcome are calculated as the number of participants completing the program as a percent 
of all program participants starting the program. 


Setting an outcome in the planning phase can be worrisome, but it is critical to a strong proposal, 
plan and program. All too often there is the desire to refrain from clearly articulating expected 
outcomes for fear of not meeting them. I would suggest that thoughtfully prepared outcomes 
remove that fear as 100% perfection is an aspiration and not a reality. That being said, there is a 
way to provide expected outcomes while also acknowledging potential differences. 


For example, if a program participant begins a program but leaves shortly after the program 
begins, it may be hard to say that the program itself was at fault. In this case it may be that a 
grant program may choose to only include the participants who were still participating one 
month after starting; in practice this is called behaviorally-defined cohorts which are determined 
by participant behavior. The second outcome metric in Figure 2, calculated as the number of 
participants completing the program as a percent of all program participants one month after 
starting the program, is an example of a metric based upon a behaviorally defined cohort. 


The American Association of Community Colleges, in the development of their Voluntary 
Framework of Accountability (www.aacc.nche.edu/vfa), allows for the recognition that students 
use community colleges in a number of ways, including the acquisition of credits to apply at 
another college. As such, they have developed a behaviorally-defined cohort where students are 
only counted after they have enrolled for 12 credit hours. 


Process Metrics. Process metrics are those interim measures that are connected, or statistically 
correlated, to outcome metrics. Figure 2presents a number of examples for the outcomes of 
completion and employment rates. In higher education, these process metrics are the result of 
research by Ewell (2007) and Leinbach and Jenkins (2008) that identified milestone events and 
momentum points are they relate to student progression in college. 


The intent of process metrics is to monitor those activities that will lead to the outcome metrics 
of interest to external stakeholders. In some cases, such as state performance funding models, 
these process metrics are considered outcome metrics. I argue that by achieving (process) 
outcomes strongly correlated with outer (outcome) metrics one is essentially duplicating the 
same metric. Rather, the value of process metrics comes from the brainstorming and resulting 
understanding of the midpoints these measures signify. If developed with the proposal 
development and implementation team, a proposal will generate buy-in to metrics by making 
them meaningful as well as serving to support the clear expectations of practitioners and further 
reinforcing the leadership and management needed to make a grant successful as discussed later 
in this essay. 


Indicators. Indicators allow for the monitoring of program performance. More often than not, 
the indicator is framed as a comparison of one program to another. The end result of such a peer 
approach is often statements along the lines of, “Our program had outcomes comparable to other 
similar programs.” I am of the opinion that such peer comparisons are meaningless and 
downright detrimental to program performance because validating performance by comparing to 
others puts in place a mindset of acceptable failure. We have all heard others say something 
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along the lines of “that’s not bad for this population we are serving.” 


Put simply, the indicators for your program should be levels of performance you expect to 
achieve as a result of implementing the program for the population your program is serving as 
compared to the program’s actual performance, not another program. Indicators, developed 
during grant development like all metrics and expected levels of performance, must be clearly set 
and tracked, compared to actual performance and used to improve the program — not to explain 
away a lack of performance. The following three sections illustrate this approach in greater 
detail. 


Part 2: Requiring Projections 


Practitioners need to recognize that funding has associated expectations. Increasingly, in the 
current and likely future competitive environment for funding, “blank checks” are going to 
increasingly diminish. This means that outcome metrics, process metrics and indicators must be 
detailed as part of the proposal, prior to implementation. 


As an example, if the program believes that weekly attendance rates impact program completion 
rates, then attendance rates should be projected at regular intervals. It is reasonable to expect 
that, since attendees are humans, participants may miss an activity on occasion — especially as 
grant-funded programs tend to focus on students who may be at greater risk than the rest of the 
population. So, it would be appropriate for a program to set expected activity attendance rates at 
a certain percent. These projections must be a part of the proposal development process and as 
importantly accurately measured throughout the program’s implementation. Doing so will allow 
for the comparison of data to improve the program and support the completion of the program. 


Part 3: Comparing Data to Inform Mid-Course Improvements 


Comparing the data that was projected to actual data is a stronger indicator of performance than 
it would be to attendance in another program because of difference in program delivery such as 
time of day, duration, quality of experience, the availability of transportation, and other factors. 


Continuing the discussion, let’s imagine that a grant program consisted of a series of 50 
activities. Further, it was projected — prior to the implementation of the program — that the 
attendance rate for each program started at 100% for the first 3 activities and then leveled out at 
94% starting at the 16" activity. 


After the first 10 activities, the program the grant manager looks at the actual data as compared 
to the projected data with the implementation team to see if mid-course adjustments need to be 
made. Figure 3 illustrates the outcomes to date. 
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The implementation team then examines the data to find that attendance is less than expected and 
will likely impact the number of participants who will complete the program. The team then can 
begin to use the data for program improvements rather than waiting until the end of the 50-week 
period, at which point it is too late to take action. These actions may include contacting students 
that have been absent to understand why they are missing activities or disaggregating data to see 
if particular sub-populations are absent more than other sub-populations. One can imagine that, 
upon getting more information, the program offers a catch-up day where previously offered 
activities are presented, individual circumstances are overcome, or delivery is reimagined to 
engage students in a meaningful manner--the end result being that more students attend activities 
and complete the program. 


Part 4: Using Data to Explain Performance and Highlight 
Program Improvement 


By considering how outcome metrics, process metrics, and indicators are implemented 
throughout the duration of the program prior to its implementation, a clear direction for all 
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involved stakeholders may be achieved. It will also result in substantive and meaningful interim 
reports and final reports that use data to drive the narrative rather than artifacts of simple 
statements of compliance coupled with justifications that other programs may be performing 
similarly. Review the following two program updates and see if you agree. 


e “Our completion rates are comparable to similar programs.” 


e “Our completion rates were higher than we initially projected. We believe a contributing 
reason why is that we tracked attendance rates and found that initially they were below 
what we expected. However, after reviewing the data 10 activities into implementation 
we made some adjustments that included revamping our delivery and engaging 
workforce partners to make the program more applied and meaningful to participants 
who were able to see real worked connections. The result was that our attendance rates 
climbed to 96%, which we believe impacted our completion rate.” 


The second update provides a richer understanding of the program to the reader; it also makes 
writing the reports easier, and subsequently funders more engaged and likely to be supportive in 
those instances where the actual outcomes do not meet projected outcomes. You may also notice 
that the second update included examples of how the data was used to make changes to program 
delivery. In almost all cases I am aware of, grantees are requested to provide lessons learned 
from the program so that funders can, like grantees, improve. Admittedly, this approach is 
different, requires thought, and may be threatening. To implement it well takes a grant manager 
who can both lead and manage. 


The implementation of this process relies not only on the structures suggested in this essay, but 
practitioners with the management and leadership skills necessary to implement change; 
something that is no small feat itself. Rather than give an extensive treatise on leadership and 
management, I share a few thoughts about each as they relate to grant programs from my 
experience as both a grantee and the manager of a multi-million dollar grant program. 


Leading the Program. If the proposal is developed with implementation in mind, then the 
grant manager has the vision set for them. The act of leading, then, is generating the interest and 
belief in the grant vision set before them. It requires the grant manager to have a firm 
commitment to the proposal, however, as the fidelity of implementation has long been 
recognized as a barrier to carrying-out programs. Staff are also watching grant leaders. 


I once had the opportunity to conduct a site visit where a project staff member expressed 
appreciation that the project was being held accountable for the actions delineated in the 
proposal. In the same breath, the staff member shared that the grant leader informed staff that 
they did not expect to meet all of the proposed activities. This is exactly the lack of commitment 
by leadership that leads projects to fail, often at taxpayer expense. Conversely, there may be a 
situation where staff are not implementing the proposed actions as planned; this requires firm but 
fair management. 


Managing the Program, Not the Staff. Being elevated to the position of leadership is nice. 
It is empowering. However, while it requires weaving together disparate pieces into one it also 
requires one assume the responsibility to manage. And management is not nearly as pleasant. 


78 


Management requires one to set and implement a clear direction, to navigate unexpected 
challenges by relying on the plan not personal opinions, to facilitate tough conversations, and to 
make decisions that are best for the program rather than staff. All too often decisions are 
indefinitely delayed, and yet what is often unrecognized is that not making a decision — or 
inaction — is actually a decision. 


In short, grant management means managing the grant, rather than the staff itself. By doing so, 
one can remove the personal aspect as decisions are made in terms of what is best to meet the 
objectives, not the wishes of staff. 


Moving Forward. In this essay I shared my perspective on how grant programs can be 
developed with an eye towards improvement. It requires leadership, thoughtful work throughout 
all aspects of the program, and effective management. This level of commitment makes one 
vulnerable if outcomes are not met, but there is no other way to lead. Receiving a grant and the 
funds that accompany it is more than a mark of distinction for the institution and leader. It is 
more than an on-ramp to further funding. It is a commitment to deliver on what is proposed. 
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CREATING AND SHARING KNOWLEDGE THROUGH 


COMMUNITIES OF PRACTICE sy oscar retix 


Abstract 


Building a community around 
a shared learning interest is 
not da new or _ innovative 
concept. However, collective 
learning and __ information 
sharing can bring new 
perspectives and improved 
practices to a_ professional 
organization in need of wider 
member participation. 
Communities of Practice in the 
context of College Access and 
Success Programs provide 
community members with 
additional resources, tools, 
and research findings to 
improve student services and 
outcomes. 
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Introduction 


Creating a community in which members can share 
knowledge enhances the quality of services to students. 
Practitioners benefit by receiving timely and specific 
knowledge that is needed to better advise and motivate 
students. Whether seeking research-based practices or 
logistical information, the over-burdened college access 
and success professional can benefit through engagement 
in a community of peers who share a common interest. 
A Community of Practice can simultaneously engage 
such professionals in promoting research endeavors, 
implementing research-based practices, and creating 
opportunities to have robust discussions on what is most 
beneficial to their students and programs. Learning that 
happens in a community of practice is not separate from 
professional context and identity. In this essay, I share 
how my experience as a student shaped my professional 
life and how a Community of Practice can be a way to 
improve our work by presenting benefits of a community 
of practice for college access and success professionals. 


My Experience 


Communities of Practice can have a profound and direct 
impact on student success. Research shows that a study 
abroad experience, for example, can make a significant 
impact on first-generation students. As I recall the most 


meaningful co-curricular opportunity of my undergraduate experience, I know now how 
fortunate I was to participate in a study abroad program in London for 12 weeks. As a Pell Grant 
recipient in the 1980’s, I was part of a small number of low-income, first-generation students 
who studied abroad. Living and learning in London was the ultimate academic and collegiate 
experience of my undergraduate years. I learned living history and political science from 
students and professors from South Africa, Iraq, Iran, and Saudi Arabia. We took day trips to 
Oxford, Windsor Castle, and Parliament, and weekend trips to Scotland and Amsterdam. At the 
end of three months, I was forever changed. And my new world perspective provided a global 
foundation for future work, professional confidence, and personal perspectives. 
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Empirical evidence validates my positive study abroad experience translating to success during 
my college years and beyond. Studying abroad is a type of transformational experience that has 
a strong, positive impact on student college academic performance and graduation. This rings 
true for me. I finished my undergraduate studies with high academic performance in my last 
semester, and successfully transitioned to my professional life with more confidence and a 
greater understanding of the world. I now am passionate about providing international study 
opportunities to other Pell-eligible and first-generation students. 


My positive experience with international education, validated by research on its benefits, now 
drives my personal interest in being part of a community of practice that promotes the value of 
studying abroad, and provides a roadmap to make it happen. College Access and Success 
practitioners connected to an International Education Community of Practice, for example, can 
significantly increase opportunities for participation in activities that greatly benefit students. 
Other communities of practice can be developed based on interests driven by subjective 
preferences, and yet substantiated by empirical evidence on its benefits. 


Knowing the powerful impact of co-curricular experiences such as study abroad, what can be 
done to expand these opportunities to more low-income, first generation students? Most critical 
is that College Access and Success staff have timely, useful, and accurate information to share 
with and motivate students to participate in international experiences. There are a myriad of 
technical and logistical questions to be answered, and staff need reliable and current sources of 
information to encourage a student’s initial commitment and to take the initial steps in this 
complex process. The most helpful information will be based on sound practices that have been 
subject to research and evaluation processes. Each student situation is unique, and some students 
may require additional support, particularly with families reluctant to support their students 
traveling beyond the United States. 


What is a Community of Practice? 


As the name implies, Communities of Practice are created to support daily practice through 
exchange and dialogue with other college access and success professionals. Sharing a craft 
and/or a profession forms the basis for a group to evolve naturally because of the members’ 
common interest in a particular domain or area. While the technical support and content 
knowledge are clearly important, the learning process is reinforced through networking, dialogue 
and sharing of effective practices, and thereby helping each other grow personally and 
professionally (Wenger, 1999). The concept of academic and professional learning communities 
has been used for years by K-12 schools to improve teaching and learning. To better support 
College Access and Success staff in their daily practice, Communities of Practice can be formed 
around different topics. Below, I discuss benefits of community of practice for college access 
and success professionals, and how I went about creating a community of practice. 


Benefits of Communities of Practice 


In the College Access and Success profession, our common interest is college degree attainment 
for low-income, first-generation students and other student groups underrepresented in higher 
education. To achieve this common goal, critical interim steps, such as secondary school success 
and college enrollment and persistence, must be supported. Yet, even with a robust training 
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curriculum for College Access and Success staff, there are gaps in knowledge about effective 
practices. When staff participates in learning communities where information and experiences 
are shared, they simultaneously increase their own knowledge base and engage with a supporting 
community. 


Timely Information on Demand. A basic function of a Community of Practice is to 
disseminate timely information for College Access and Success professionals to provide better 
advising to secondary school students on topics such as early college planning, financial aid, 
academic support, and the critical transition from high school to college. Financial aid processes 
change every year, and these updates are critical to receiving the most financial support possible. 
For college success staff, information and resources on topics, such as personal identity, social 
justice, engagement in undergraduate research, and other high-impact experiences, can support 
their efforts to increase college retention. Communities of Practice facilitate information sharing 
from practice and closing gaps in knowledge. This additional source of professional learning 
improves the overall competency of staff as they seek to support students to graduation. 


While it may be true all information needed already exists on the Internet, the benefit of access to 
relevant information when planning a complex task can make a critical difference. When 
visiting a new city, getting timely and accurate information about a particular activity is 
priceless. As a stranger to a new location, it would take considerable effort and time to 
personally find all the information about restaurants, activities, hotels, etc. A clear, tangible 
benefit of a Community of Practice is the mapping of information in a way that is readily 
available and easily accessible for college access and success professionals. Reflecting back to 
my London experience as a student, I would have benefitted from the support of a well-informed 
advisor as I was preparing for my study abroad experience. 


I was fortunate to attend a small, private university that was truly an international university. 
Yet, as a first-generation college student, I could have easily been challenged by a process that 
entails multiple, complex steps. The benefits I received as a student in a well-informed setting 
also applies to college access and success professionals. For example, if a student is interested in 
undergraduate research and if a professional has never worked to provide such information to 
students, the professional needs to find out what the basic information would be most helpful to 
that student How can this information be readily available to the entire College Access and 
Success community of practitioners? A Community of Practice is structured to facilitate timely 
and relevant information sharing through communication and social media platforms. 


A Sustained Advocacy for High Impact Academic Activities. One area in particular 
where Communities of Practice can be beneficial is encouraging students to participate in high- 
impact academic and co-curricular activities (Kuh, Kinzie, Schuh, & Whitt, 2010). Studies show 
students engaged in such activities have significantly higher success rates than students who do 
not participate in such experiences. Some of these high-impact experiences include first-year 
seminars, learning communities, undergraduate research activities, internships, capstone courses 
and projects, and, as mentioned earlier, study abroad. The Communities of Practice, through 
their sustained engagement of College Access and Success professionals, can map and provide 
access to the wide array of information needed to effectively implement a high-impact service or 
activity. 


82 


Opportunity for Rising Leaders. The Communities of Practice also provide opportunities 
for potential and rising leaders, who typically do not attend national conferences, to be involved 
in leadership roles beyond their state and regional associations. An individual in an advisor or 
coordinator role would have the opportunity to become a Community of Practice chair, co-chair, 
or subcommittee chair and engage with constituents electronically and through social media. 
These leadership roles could leverage support from institutions to attend the national, state, and 
regional association conferences. Rising leaders can also build their professional and academic 
portfolio through involvement in research and evaluation practices, leading to improved project 
outcomes and increased legitimacy for the profession. Communities of Practice also build 
confidence, communication, and information sharing skills for emerging professionals. 


Creating Community of Practice of TRIO Professionals 


How to Create a Community of Practice. As practitioners, we have insights about our own 
capacity to provide substantive advice and support for students beyond basic program-prescribed 
services. We know our limits yet we want to do more. I have come to realize that guiding 
students towards and through high-impact practices such as study abroad is a journey identical to 
the college enrollment path. Along with other practitioners with similar interests, the 
International Access Community of Practice (CoP) was created to provide colleagues with 
resources and information on funding opportunities, necessary application and documentation, 
and other key resources needed to help student navigate the process. Money, aspiration, and 
information--the more I can provide to students, the closer we get to the finish line. 


As College Access and Success professionals identify the need and benefits of a particular CoP, 
professional organizations, such as the Council for Opportunity in Education, can provide the 
structural and logistical support for practitioners across the nation to communicate and share 
relevant content and resources. While the International Access CoP has a specific area of focus, 
other Communities of Practice can be developed by practitioners who identify a particular area 
of interest. 


Use of Annual Conference. Annual conferences provide a time and place for members of the 
Communities of Practice to meet, engage, share information, provide mutual support, and create 
a community based on shared interests. The leaders of the Communities of Practice facilitate 
discussion on current trends, ways to disseminate information, and ways to identify gaps in the 
knowledge needed to support its members. The conference meetings also reinforce the need for 
greater collaboration and mutual support. 


Use of Electronic Communication to Sustain Engagement. Communication between 
meetings at national conferences is sustained through the use of electronic communication and 
social media. Numerous web-based tools for dissemination of information and community 
engagement are easy to use and free of cost. Traditional electronic methods include email 
listservs, electronic newsletters, shared document sites, websites, and links to resources. 
Communication through social media can include Facebook, Twitter, LinkedIn, Pinterest, and 
Google Plus. For professionals already engaged in social media, this approach of 
communication will be familiar, and will focus on distinct topics that support their practice. 
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In the Fall of 2015, the Council for Opportunity in Education, a membership organization of 
college access and success professionals, implemented its first three Communities of Practice, 
including the International Access CoP. This CoP provides support and information to 
practitioners on how to guide students through the complex process of an international 
educational experience. As I mentioned above, the CoP utilizes electronic communication to 
share learning from high-impact activities and to connect professionals working on similar issues 
to access to timely suggestions. The STEM CoP provides a forum for effective practices on 
supporting underrepresented students in the sciences to undertake and succeed in the STEM 
fields. The Research, Evaluation and Data Use CoP serves to encourage College Access and 
Success practitioners to implement practices that are based on sound research and evaluation. 
This CoP also serves as a forum to encourage new research activity, connect with other research 
and evaluation communities, and identify resources and points of collaboration and 
dissemination of effective practices. Encouraging a culture of inquiry, assessment, and shared 
knowledge in the College Access and Success community can improve program effectiveness, 
and cultivates legitimacy in arenas where student outcomes shape education policy that drive the 
future of the profession. 


Reflections and Conclusion 


The Community of Practice model offers added value to professional organizations striving for 
greater membership engagement, community building, and leadership development. The 
development of these communities will be an organic process in which professionals respond to 
an unaddressed issue or tangible benefit in their practice. Ultimately, the success of the 
Communities of Practice will depend largely on how professional staff perceives its value. 
Organizations supporting College Access and Success efforts can reinforce their mission by 
providing professional staff with critical and timely information and support to improve their 
practice, leading to better outcomes for their programs. 
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UNDER THE PRESSURE TO REPLICATE “EVIDENCE- 


BASED” INTERVENTION, TRIO PROGRAMS SHOULD 
BUILD THEIR LOGICS OF EVIDENCE USE sy mika yvamasuira 


Introduction 


In 2014, when TRIO grantees saw the instructions for 
the grant application for the new grant cycle of Student 
Support Services (SSS), they saw a new requirement. 
Notice of Inviting Applications for Student Support 
Services (SSS) issued for the 2015 Competition included 
a Competitive Preference Points (CPP), which states the 
applicants can earn up to six points by proposing two 
interventions that are “supported by moderate evidence 
of effectiveness” (79. Fed. Reg. 2014, page 79575). To 
show that their proposed intervention is_ effective, 
applicants had to submit one supporting study that meets 
What Works’ Clearinghouse (WWC)’s moderate 
evidence standards. The similar requirement appeared in 
2016 in the Talent Search grant competition. To earn six 
points, applicants had to demonstrate that each of 
proposed “strategy is based on research that meets the 
Moderate Evidence of Effectiveness standard” (80. Fed. 
Reg. 2015, page 79575) and “demonstrate how the 
proposed project activities align with the cited study with 
sufficient fidelity” (80. Fed.Reg. 2015, p. 79575). It 
appears the competitive preference points (CPP) resulted 
in enforcement because the cut-off score of SSS grant 
application was around 104, indicating that no new 
program that did not have any prior experience could 
receive funding without earning the competitive 
preference points. 


As a researcher for the Pell Institute, a research arm of 
the Council for Opportunity in Education (COE), a 
membership organization of TRIO and other college 
access and success programs, I have heard concerns 
about TRIO program evidence standards raised by COE 
and Pell staff and TRIO program directors and staff. 
These concerns include the requirement to replicate 
interventions that were developed somewhere else could 
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Abstract 


This essay discusses the origin of the 
competitive preference priorities 
(CPP) that appeared in Student 
Support Services (SSS) and Talent 
Search grant applications in 2014 and 
2015. In these competitions, CPP 
encouraged programs to propose 
evidence-based _ interventions. I 
discuss the CPP as a series of policy 
instruments that the federal 
government implemented to promote 
evidence-based policymaking. The 
idea underlying CPP is that by 
encouraging grantees to implement 
evidence-based interventions, _ the 
grant programs are likely to deliver 
results. One important element of this 
assumption is that randomized 
controlled trials (RCTs) are the best 
methods to ascertain if an intervention 
is effective. After describing 
limitations of applying findings from 
RCTs to improve program practice, I 
argue that TRIO programs need to 
articulate their own specific logics of 
evidence use that reflect how TRIO 
professionals work in practice. This 
is because RCTs alone can provide 
very limited information about actual 
interventions for TRIO professionals 
to improve their own programs. 
Research questions that reflect how 
TRIO programs actually work are 
needed. 


discourage innovation among programs and whether or not there are sufficient interventions that 
meet WWC standards that can be implemented by SSS programs. They also cited challenges in 
coordinating the evidence-based services that are required by legislation within the limited 
program budget. The overall sentiment expressed was that CPP is another hoop TRIO programs 
must jump through to receive federal funding. 


Like many federal education programs, TRIO programs are under pressure to show evidence that 
they are effective. The programs have been regularly monitored, and grantees must meet 
performance objectives. Prior experience points, which programs earn by meeting a set of 
performance objectives created at the beginning of each grant cycle, make up a significant 
portion of the grant scoring. This means a program must meet a set of performance objectives in 
order to be funded again. 


In this essay, based on my reading of publicly-available documents, I discuss the CPPs as a part 
of policy instruments the federal government has been using to promote evidence-based 
policymaking. The government views requiring evidence-based intervention as one of their 
effective grant-making strategies. I also argue that the CPP reflects the idiosyncrasy of 
government policy making, where one aspect of idea is taken up and other aspects are 
marginalized or lost. By describing limitations of the assumptions underlying the CPPs, I 
discuss why it is critical for TRIO programs to identify a research agenda that reflects the actual 
work of TRIO professionals. While TRIO programs may view CPP as another hoop to jump 
through, it is important to understand the big picture because the CPP carries certain expectations 
about how TRIO programs should improve. The primary audience of this essay is TRIO 
professionals. 


Origins of Competitive Preference Points Attached to 
Evidence-Based Interventions 


The Education Department General Administrative Regulations (EDGAR), which governs grant 
programs, provides a legal justification of inserting CPPs in the most recent SSS and Talent 
Search grant competitions. Recent revision of EDGAR in 2013 is in line with the government- 
wide initiative to infuse evidence use in government grant making (McNail, 2013). The Office 
of Management and Budget (OMB) has issued guidance documents to “better integrate evidence 
and rigorous evaluation in budget, management, operational and policy decisions” and it 
recommended agencies to adopt “more evidence based structures for grant programs” (OMB, no 
date). A document that provided guidance on Fiscal Year (FY) 2014 budget outlines “evidence- 
based grants” (OMB, May 18, 2012, page 2), and one of the approaches recommended by OMB 
is as follows: 


agencies can provide points or significant competitive preference to 
programs that the agency determines are backed by strong evidence, 
and can build the evidence base by embedding evaluation into 
programs. (OMB, May 18, 2012, p. 3) 


As described in “Strategic Plan for Fiscal Year 2014-2018” and in the most recent “FY16-17 
Agency Priority Goals,” the Department of Education continues to use competitive preference 
points on discretional grant programs to “enable evidence-based decision making” (U.S. 
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Department of Education, 2014, 2016). The Department of Education estimates the percentage of 
new competitive funding that supported evidence-based projects (including strong, moderate and 
evidence of promise as defined by What Works Clearinghouse) in FY 15 was 16 percent, and it 
aims to increase it to 20 percent by September 30, 2017 (U.S. Department of Education, no date). 
TRIO-Talent Search is listed as one of the contributing programs, along with “various Higher 
Education Act Title III programs” and” Investing in Innovation Fund (i3)” for FY 16-17(U.S. 
Department of Education, 2016). 


The Obama Administration used grant-making as a strategy to promote evidence use to drive 
outcomes (Berlin, 2016; Haskins & Margolis, 2015; Nussele & Orszag, 2015). Based on 
interviews with over 130 individuals who were involved in the Obama Administration, Haskins 
and Margolis (2015) describe how the administration moved from formula grant-making to 
competitive grant-making. They report “the administration wanted to take advantage of the 
pressure created by competition to spend the money on projects that meet strong evidence 
criteria” (pp.29-30) and the administration assumed that competition helps to ensure “that only 
the highest quality applicants win funding” (p.30). The Administration viewed it as “the smarter 
investment” to invest in programs that can “deliver results” (Orszag, 2009). 


The Obama Administration introduced tiered evidence, where programs that have “strong 
evidence” received more funding than programs that have “some evidence” in its signature social 
programs, such as Home Visitation Programs at the Department of Health and Human Services 
and Investing in Innovation (13) at the Department of Education. This approach was to ensure 
that funded programs would deliver results and to promote innovation, as OMB described below: 


For these two very different subjects (referring to Home Visitation 
Program and Teen Pregnancy Prevention Program), we are using a 
similar, two-tiered approach. First we’re providing more money to 
programs that generate results backed up by strong evidence. That’s 
the top tier. Then, for an additional group of programs, with some 
supportive evidence but no as much, we’ve said: Let’s try those too, but 
rigorously evaluate them and see whether they work. Over time, we 
hope that some of those programs will move into the top tier—but, if 
not, we'll redirect their funds to other, more promising efforts. This 
design differs from the typical approach. We haven't simply created a 
block grant and told states they can do whatever they want, nor have 
we dictated a particular program design and told everyone to follow it. 
Instead, we’ve said that we are flexible about the details of the 
program; we only insist that most of the money go toward the programs 
backed by the best available evidence, and the rest to programs that 
are promising and willing to test their mettle. (Orzag, June 8, 2009) 


Other grant-making strategies OMB has recommended include use of evidence in formula grants 
and Pay for Success (OMB, May 18, 2012). Pay for Success forces grantees, often intermediary 
organizations, to have significant evidence that proposed programs will deliver results, as 
program success determines if applicants receive payment from the government. As it is, the 
CPPs included in TRIO competitions are a part of the grant-making strategy that was intended to 
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promote evidence-based policymaking in the federal government. The underlying assumption is 
by investing in programs that work, the government and taxpayers will get better results. 


Spread of Evidence-Based Programs/Interventions 


While the competitive preference priorities (CPPs) attached to evidence-based interventions was 
new for TRIO programs, there were grant programs with similar requirements prior to the 
Obama Administration. For example, previously “evidence-based” appears in laws related to 
social welfare programs, such as immunization, drug prevention, and care for mothers with new 
born babies, where grantees were to implement “‘evidence-based” assessment and practices. In 
education, in the late 1990s, Safe and Drug Free programs issued Principles of Effectiveness, 
which states programs should be “guided by research and best practices.” This guidance was 
later incorporated in the No Child Left Behind Act in 2001 (Weiss, Murphy-Graham & 
Birkeland, 2005). The Department of Education’s Safe and Drug-Free School Program gathered 
a panel of experts to create an evidence-based programs list. The panel reviewed programs on 
four criteria: efficacy, quality, educational significance, and usefulness to others, and compiled a 
list of exemplary and promising programs (Department of Education, 2001). 


Subsequently, the federal government began initiatives to identify effective interventions, which 
created inventories of programs that are evidence-based. What Works Clearinghouse (WWC), 
created in 2002, is one of them. Some others are: 


e Evidence —Based Practice Centers (EPC) by Agency for Healthcare Research and Quality 
(AHRQ), created in 1997, focuses on evidence on relative benefits and risks of healthcare 
interventions. 


e The Guide to Community Prevention Services by Health and Human Services (HHS), 
created in 1996, provides evidence-based recommendations and findings about public 
health interventions and policies to improve health and promote safety. 


e National Registry of Evidence Based Programs and Practices (NREPP) by the Substance 
Abuse and Mental Health Services Administration (SAMHSA), created in 1997, provides 
for the scientific basis and practicality of interventions that prevent or treat mental health 
and substance abuse disorders. (Government Accountability Office, 2009) 


These evidence review initiatives were to provide information to professionals, policymakers, 
and researchers about effective interventions by reviewing studies. Each initiative created its 
own evidence standards, reporting template, and review guidelines. According to a report 
published by the Government Accountability Office (GAO) in 2009, the review process was in 
general similar across the initiatives, but there were some variances. While randomized 
controlled trials (RCTs) and quasi-experimental studies are the most commonly reviewed study 
designs across the initiatives, Evidence-Based Practice Centers (AHRQ) and Guide to 
Community Preventive Services (CDC) also review observational studies, such as time series 
and case control studies. While the What Works Clearinghouse (WWC) did not assess 
implementation fidelity, four other initiatives explicitly assess intervention fidelity “through 
either describing in detail the intervention’s components or measuring participants’ level of 
exposure” (GAO, 2009, p. 15). 
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These resources, procedures, and experiences of identifying, reviewing, and rating studies and 
interventions since the late 90s contributed to the Obama Administration’s evidence-based 
initiatives. 


Spread of the Idea of Causality as Effectiveness 


The underlying idea of attaching competitive preference points (CPP) is that the programs will 
produce better results by implementing evidence-based interventions. Evidence standards set by 
WWC determine if an intervention is evidence-based because the study design must meet 
WWC’s design standards and also show positive results. According to the WWC, only RCTs 
potentially meet design standards without reservation and certain RCTs and quasi-experimental 
studies with comparison groups potentially meet WWC design standards with reservations (What 
Works Clearinghouse, 2014). According to the Notice of Inviting Applicants of the SSS grant, 
the study has to show “statistically significant favorable impact” (79. Fed. Reg. 2014, page 
75724). This view of effectiveness as determined by causal relationship between an intervention 
and outcomes has been in place for over a decade in the Department of Education and the federal 
government at large. 


The Institute of Education Sciences (IES) established What Works Clearinghouse (WWC) in 
2002. The goal of WWC is: 


to be a resource for informed educational decision making. To reach 
this goal, the WWC identifies studies that provide credible and reliable 
evidence of the effectiveness of a given practice, program or policy 
(referred to as “interventions”’). (WWC, no date) 


WWC defines “effectiveness” as follows: 


an intervention demonstrates effectiveness if the research has shown 
that it caused an improvement in outcomes. (WWC, no date) 


The Department of Education promoted this idea of effectiveness since early 2000. IES Director 
Grover (Ross) Whitehurst’s presentation at the American Educational Research Association 
(AERA), after the passage of Education Science Reform Act of 2002, which created the Institute 
of Education Sciences (IES), states IES’s position: “Randomized trials are the only sure method 
for determining the effectiveness of education programs and practices” (Institute of Education 
Sciences, no date). 


A report by Congressional Research Service (CRS) published in 2006 illustrates the spread of the 
idea that RCTs is the best way to determine program effectiveness within the Department of 
Education and at the Office of Budget and Management (OMB). The Department of Education 
published “Notice of Final Priority” in November 2003 to clarify “scientifically based research” 
stated in the No Child Left Behind Act (NCLB) and to make it a priority to be used for all 
programs beyond those funded by NCLB, which became effective in February, 2005. The final 
notice states: 


The definition of scientifically based research in section 9201(37) of 
NCLB includes other research designs in addition to the random 
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assignment and quasi-experimental designs that are the subject of this 
priority. However, the Secretary considers random assignment and 
quasi-experimental designs to be the most rigorous methods to address 
the question of project effectiveness. While this action is of particular 
importance for programs authorized by NCLB, it is also an important 
tool for other programs and, for this reason, is being established for all 
Department programs. Establishing the priority on a Department-wide 
basis will permit any office to use the priority for a program for which 
it is appropriate. (70. Fed. Reg. January 25, 2005, p. 3586) 


A similar message appeared in another publication by the Department of Education, “Identifying 
and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly 
Guide,” which was to “provide educational practitioners with user friendly tools to distinguish 
practices supported by rigorous evidence from those that are not” (Brass, Nunez-Neto & 
Williams, March 7, 2006, page iii). CRS summarizes: 


guidance asserted that evaluation methods other than RCT and certain 
quasi experiments (1) have no meaningful evidence’ to contribute to 
establishing whether an intervention was ‘effective’ and (2) cannot be 
considered ‘scientifically-rigorous evidence’ or ‘rigorous evidence’ to 
support using an educational practice to ‘improve educational and life 
outcomes for children. (Brass, Nunez-Neto & Williams, March 7, 2006, 
p. 25) 


The Office of Management and Budget (OMB) also promoted this conceptualization of program 
effectiveness during Bush Administration with the technical assistance from the Coalition for 
Evidence-Based Policy, a private non-profit organization (Brass, Nunez-Neto & Williams, 
March 7, 2006, Coalition for Evidence Based Policy, no date). “What Constitutes Strong 
Evidence of a Program’s Effectiveness” in 2004 is a guidance document to support agencies with 
reporting on The Program Assessment Rating Tool (PART), a set of questions asked to agencies 
to draw conclusions about program benefits and recommendations to improve programs. PART 
asked if an evaluation conducted for its program is sufficient in scope and quality and if the 
evaluation indicates the program is effective. The guidance document states: 


The most significant aspect of program effectiveness is impact—the 
outcome of the program, which otherwise would not have occurred 
without the program intervention. While it is feasible to measure the 
impact of the program, RCTs are generally the highest quality, 
unbiased evaluation to demonstrate the actual impact of the program. 
(Office of Management and Budget, 2004) 


Similarly, a guidance document for FY 2007 PART reporting describes: 


most significant aspect of program effectiveness in impact—the 
outcome of the program, which otherwise would not have occurred 
without the program intervention. A number of evaluation 
methodologies are available to measure the effectiveness of programs. 
Some, such as randomized controlled trials, are particularly well suited 


go 


to measuring impacts. .... Quasi-experimental studies should be 
scrutinized given the increased possibility of an erroneous conclusion. 
(Office of Management and Budget, January 29, 2007, pp 30-31) 


The promotion of these specific conceptualizations of “effectiveness” and “evidence” by the 
executive branch has drawn attention from legislative offices. Over a decade ago, the 
Congressional Research Service (CRS) produced a report that describes design features of RCTs 
and potential issues the Congress might encounter in the future (Brass, Nunez-Neto & Williams, 
March 7, 2006). GAO’s report in 2009 on the Coalition for Evidence Based Policy’s Top Tier 
Evidence Initiative, which identifies interventions that meet Coalition’s top tier evidence 
standards, compared the Coalition’s review process with the review process taken by six 
federally-funded evidence review initiatives, such as What Works Clearinghouse (WWC) and 
National Registry of Evidence Based Programs and Practices (GAO, 2009). These reports 
presented limitations of the RCTs’ research design and for making policy decisions. For 
example, one of the conclusions by GAO report was: 


Requiring evidence from randomized studies as sole proof of 
effectiveness will likely exclude many potentially effective and 
worthwhile practices. (GAO, 2009, cover page) 


Prior to these reports, the research community raised concerns about a specifying research design 
for the purpose of improving knowledge about education because scientific inquiry uses multiple 
types of research design. In 2001, during the time Education Science Reform Act was in the 
Congress, the National Research Council gathered a Committee on Scientific Principles for 
Education Research, which produced guiding principles for scientific inquiry. In the report, after 
acknowledging that a part of the Committee charge was to provide recommendations to improve 
education research, the report states: 


attempting to boost the scientific basis of federally funded education 
research by mandating a list of “valid” scientific method is a 
problematic strategy. (National Research Council, 2002, p. 130) 


It appears these concerns were not reflected in the guidance and regulations developed by the 
federal government. It may be possible that policy makers i.e., government officials, legislative 
staff, and advocacy groups who were involved in policymaking, limited their frame of reference 
to what they saw as manageable (National Research Council, 2002). From reading Haskins and 
Magolis (2015)’s report on how Obama Administration’s transition team, government officials, 
legislative staff, and advocacy groups crafted evidence-based initiatives and how grant 
competitions and selection criteria were developed, it seems the notion of effectiveness equals 
causal relationship had been institutionalized. Disagreements discussed in their report centered 
around whether evidence-based programs should be limited to programs supported by RCTs only 
or should be expanded to programs supported by quasi-experimental studies, as it determines 
which model programs would be replicated by grantees. It appears there was no question about 
the assumption that replicating evidence based intervention is most likely to deliver results. 
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Limitations of Evidence-Based Interventions 


TRIO programs raised concerns when they learned about the competitive preference points 
(CPP). The concerns derive from the limitations of RCTs for informing practice, which I 
describe below. 


Lack of External Validity. Even when RCTs are well-designed and well-implemented, they 
are limited in external validity. An intervention that showed positive results in one site does not 
guarantee the intervention will produce the same effect in other sites. The study tells the 
intervention worked in a certain context in which the experiment happened. Sometimes grant 
programs assign higher scores to interventions that are found to be effective in multiple sites. 
For example, the Corporation for National and Community Services’ AmeriCorps grant 
competition provided more points to a proposal that proposed an intervention that had been 
tested nationally or in multi sites, while if applicant submitted a study that was conducted in one 
site, they received lower points (Corporation for National and Community Services, 2015). 
However, as of now, there are not many replication studies in education. The Department of 
Education asked SSS applicants to submit one study that includes “a sample that overlaps with 
the populations or settings” that meets the definition of moderate evidence of effectiveness; 
however, there was no further definition of what it means by “population” and “settings.” 


Another difficulty surrounding multiple replication studies seems to be that programs improve. 
The excerpt below, by a CEO of Building Educated Leaders for Life (BELL), a program that 
provides after school and summer learning experience for high risk students, describes these 
limitations of RCTs. 


Even when we are successful generating good evidence, the story of 
what works is still incomplete. Even when we have well-conducted, 
highly rigorous studies, there are still limitations in what they tell us 
about the real nature of social problems, interventions, and changes in 
the lives of those we reach. As an example, even when randomized 
controlled trials tell us the story of an organization’s work at one point 
in time, five years later most organizations’ work has evolved. So at its 
best, good evidence is still incomplete.” 


Obtaining evidence is risky. ... So you do a randomized controlled trial, 
and it says your program works. It’s not the end of the story. It 
worked for some group of kids in some context at some point in time, 
and so we should celebrate that and make sure that’s understood. And 
your randomized-controlled-trial study two years later might say, ‘It 
doesn’t work for kids or families.’ And so there, too, we don’t want to 
unfairly punish an organization without understanding that in context. 
So this risk, I think, is something we’ll have to pay more attention to 
across sectors (Nussle & Orszag, 2015 ) 


Meta-analysis compiles results from multiple RCTs to generate an overall effect of an 
intervention implemented across different populations or settings. Systematic review could 
include other study designs and narrative synthesis to inform conditions that might have led to 
better results, such as minimum duration of intervention exposure and conditions. However, 
users of the review also need to determine the applicability of the results of these reviews (Brass, 
Nunez-Neto & Williams, March 7, 2006). For example, the guidance document by Cochrane 
Review, which conducts systematic reviews of evidence on healthcare treatments and products, 
describes the importance of readers to make informed decisions about applicability of findings of 
systematic review and meta- analysis as follows: 


Another decision users must make is whether the patients before them 
are so different from those included in the studies that they cannot use 
the result of the systematic review and meta-analysis at all..... Authors 
can sometimes help clinical decision makers by identifying important 
variation where divergence might limit the applicability of results, 
including: biologic and cultural variation, variation in adherence to an 
intervention. In addressing these issues, authors cannot be aware of, 
or address, the myriad of differences in circumstances around the 
world. They can, however, address differences of known importance to 
many people, and importantly, they should avoid assuming that other 
people’s circumstances are the same as their won in discussing the 
results and drawing conclusion. (Higgins & Green, 2011). 


Complexity of Intervention. Another limitation is that social service and education 
interventions studied are actually complex with high “causal density” (Manzi, 2010, 2012). An 
intervention studied includes many causal relationships that researchers cannot separate out. For 
example, if an after school tutoring program was an intervention, the intervention happens in a 
context that was enabled by tutors and students who may have different characteristics and 
intentions. Tutors may use various strategies, and they may have various levels of knowledge, 
including tacit understanding of how things work in a specific school, and they may approach 
their work differently. So, when this after school tutoring showed effectiveness in improving 
students’ math skills, it is difficult to know how the combinations of components led to 
effectiveness. As it is, “RCTs typically do not assess how and why impacts occur, how a 
program might be modified to improve program results.” (Brass, Nunez-Neto & Williams, 
March 7, 2006, page 16). Thus, even if the evidence-based intervention is identified, programs 
have to find ways to make the intervention work. To support implementation, some research 
reports include contact information for implementation support, logic model, theoretical 
framework, and minimum dosage; however, replication is not straightforward. 


Furthermore, because the result of an RCT stems from one intervention, an important question 
for programs is how interventions can be coordinated so the programs can produce outcomes. 


RCTs also typically do not provide a full picture of whether unintended 
consequences may have resulted from a program or indicate whether a 
study is using valid measures or concepts for judging a program’s 
success (e.g., assessing a study’s or a measure’s construct validity). 
Many of these kinds of questions have been considered to be more 


93 


appropriately addressed with observational or qualitative designs. 
(Brass, Nunez-Neto & Williams, March 7, 2006, p. 16) 


This limitation applies to research syntheses because they often focus on outcomes that are 
determined by the reviewers. Finally, there is a gap between “effective intervention” and 
decision-to-adopt intervention because “deciding to adopt an intervention involves other 
considerations in addition to effectiveness, such as cost and suitability to the local community” 
(GAO, 2009, cover page). 


Taken together, although the federal government intended to improve TRIO programs by 
implementing evidence-based interventions, because of the design limitations of RCTs, TRIO 
cannot simply replicate the interventions. The way the Competitive Preference Priority (CPP) 
policy instrument was presented to TRIO programs minimized one critical aspect that is 
necessary for delivering results, i.e., consideration of what it takes for professionals to make 
decisions on how to implement the “evidence-based” interventions. 


Need for Logics of Evidence Use in TRIO Programs 


Discussions in evidence-based practices in medicine, social services, and education have 
highlighted the complexity of professionals’ work, partly because there is a tension between 
research knowledge and practitioner knowledge when evidence-based practice policies are put in 
practice. Drawing upon this line of discussion, I argue that TRIO programs need to draft logics 
of evidence use and demand research that aligns with how TRIO professionals work. 


Over the decades, clinical knowledge has been considered to be a critical component for 
practicing evidence-based medicine, as the definition of evidence-based medicine below 
indicates. 


Evidence based medicine is the conscientious, explicit and judicious 
use of current best evidence in making decisions about the care of 
individual patients. The practice of evidence based medicine means 
integrating individual clinical expertise with the best available external 
clinical evidence from systematic research. (Sackett, et al, 1996, p. 71) 


In evidence-based medicine, it is widely accepted that “‘any clinician who feels restricted to 
behave only as the evidence dictates is missing the concept of evidence based medicine” 
(Stevens & Hootman, 2004, page 84), and it requires “bottom-up approach” (Sackett, et al. 1996, 
page 72). Similarly, researchers have discussed the importance of professional knowledge in 
evidence-based practice in education. For example, Davis (1999) states in his article titled, 
“What is evidence based education?” as follows: 


Establishing best practice, in both education and health care, is more 
than a matter of simply accessing, critically appraising, and 
implementing research findings. It also involves integrating such 
knowledge with professional judgment and experience (Davis, 1999, 
pp.116-7) 
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However, as exemplified in the CPP and other guidelines, there is littke acknowledgement about 
the role of professional knowledge in policies that promote evidence-based practice in education 
and social services. Policy instruments project an image that professionals should replicate 
evidence-based interventions. As Thomas Archibald (2016) reported in his study, there is a risk 
that this simplified view of replication is manifest in the implementation of evidence-based 
interventions in the federally funded programs. His study found status differentials of 
knowledge produced scientists and local programs, which consequently challenges the ideal of 
evidence-based intervention movement. During implementation, the program developer makes a 
tacit adaptation of interventions based on their “more privileged position as expert knowers 
while community educators feel limits placed on their agency to respond to and adapt to the 
needs and realities in their local communities” (Archibald, 2016). I argue it is important for 
TRIO programs to examine the idea and practice of “evidence-based,” because this issue 
involves “welfare of very many people, not just the egos of a few” (Scriven, 2008, p.24) 


Practitioner-Researcher Differences in Clinical Needs and Notions of Causality. To 
argue against the simplified notion of evidence-based practice, researchers have discussed that 
knowledge which comes from research and knowledge for the needs of professionals are 
different. Knowledge from external research, 1.e., RCTs or synthesis of RCTs, does not fully 
meet the knowledge needs of professionals. This line of discussion suggests that for TRIO 
programs to improve practice, they need to articulate how TRIO professionals work and they 
need to identify research questions that reflect TRIO professionals’ work. 


Burton and Chapman (2004) reviewed the concept of “causality” on which “evidence-based 
practices” hold. They argue that causality in RCTs and the causality professionals use in their 
work differ, thus, over-reliance to external clinical evidence, which derive from RCTs, does not 
reflect how health care and social services professionals make decisions about what intervention 
they should provide to their clients. They argue that theory or conception of causality by 
healthcare and social service professionals is a “generative model of causality” where “causation 
can be identified in the activation of an internal potential of a system or substance, under 
particular conditions, to generate particular outcomes.” Thus, useful knowledge for 
professionals should “capture the complexity of the context of application via provision of an 
account of the relationships among at least the following elements: context, participants, 
practices, causal mechanisms, regularities and outcomes” (Burton and Chapman, p. 9). On the 
contrary, the concept of causality underlying evidence identified through RCTs “associationist 
model,” which “assumes causation is invisible, so the only way to establish what causes what is 
to rule out ‘spurious associations’ between independent and dependent variables.” (p. 13). 
Similar to the challenge associated with high “causal density” in an intervention, the external 
clinical evidence does not necessarily provide all the information professionals need to make 
causal inferences to do their work. 


Professional decision-making includes more than complex causality. Biesta (2007) argues that 
professionals sometimes do not do the most effective practice because of the consequence of one 
effect on another. Thus, “in education means and ends are not linked in a technical or external 
way but that they are related internally or constituently” (Biesta, 2007, p. 10), and value and 
ethics play part. Hammersley (2001) adds another level of complexity. Professionals do not 
always deal with an individual student, but a group of students with different levels of motivation 
and different goals, so the same action of a professional can have multiple consequences, which 
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are differentially distributed across students. Hammersley (2001) discusses conceptualizing 
educators work as a series of linear process from information to practice has a limitation 
(Hammersley, 2001). 


These discussions on how professionals make decisions explain the challenges TRIO 
professionals face when they are asked to replicate evidence-based interventions. TRIO program 
professionals must decide how to implement evidence-based interventions in relation to context, 
value, other interventions, and what is seen as best for students who are in a particular high 
school or college setting that promotes certain values and behaviors. The complexity of 
professionals’ decision making includes the fact that they have to deal with not only multiple 
causality between action and outcomes, but also causalities operating in different time frames, 
layers of context, beliefs, and values. 


Professional Learning. While the above discussion highlights what constitutes as needed 
information for professionals, discussion about how professionals improve their practice 
highlights the importance of clinical knowledge and the role of external research knowledge in 
the integration with professional knowledge. Biesta (2007) discusses, by drawing upon John 
Dewey’s conceptualization of learning, that experimentation and reflections are important part 
for professionals to change their actions. According to Dewey’s theory, professionals 
continuously build knowledge as they interact with the world. Thus, knowledge produced in the 
past, including knowledge from research and experiences (including lessons learned from trial 
and error), helps professionals by providing hypothesis for intelligent problem solving, but it 
does not tell them what they should do. For professionals, “the only way to use this knowledge 
is as an instrument for undertaking intelligent professional action” (Biesta, 2007, p. 17). Since 
problem-solving for professionals is context-specific and the context changes constantly, “for 
Dewey, professional action is not about following tried and tested recipes, but about addressing 
concrete and in a sense, always unique problems” (Beista, 2007, p. 16). If we follow this notion 
of how professionals work, professionals have to be reflective of what they know, how they 
know, and how their knowledge, belief, and actions are shaped by surroundings, and they are to 
continue to refine their working knowledge of what works and how to make it work, so they can 
take “intelligent professional action” (Beista, 2007, p.17). 


This point that professionals’ learning involves experimentation and theory building is important. 
Studies have shown that when teachers change their instructional practices, they need to make 
sense of why and how a specific instruction strategy works, and to change their strongly held 
belief of what works, they need to experience success as they try new instruction in their 
classrooms (Coburn, 2001; Spillane, Reiser & Reimer, 2002). Their sense-making shapes how 
they implement it, and the change of practice happens when research evidence is connected 
closely with the situation where they improve practice (Simons, et al., 2003). 


Burton and Chapman (2004) propose to take a notion of practitioner as theorists. They present 
exhaustive list of types of information (including research and non-research information) that 
shape various sub-decisions that lead to a decision of what intervention to provide to a client. 
They make a point that knowledge from RCT “always leave gaps” in information and 
professionals have to “fill in the gaps” by using various sources of information (Burton & 
Chapman, p. 20). Similar to Biesta’s (2007) argument about how professionals develop 
“intelligent professional action,” they argue that “ ‘evidence’ that does not engage with these 
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practical theories is likely to have little impact on practice, and hence on the outcomes 
experienced by the people served” (Burton and Chapman, p. 20). 


Varieties of Implementation of Evidence in TRIO. Based on brief conversations with 
TRIO programs, I developed the impression that TRIO programs are implementing evidence- 
based interventions in various ways. There are theories programs used to select a specific 
evidence-based intervention and decide when and how to replicate the evidence-based 
interventions in their programs. I think it is important for TRIO programs to document their 
theories and identify questions that will help them to build and refine these theories. I think 
TRIO programs should demand these questions be researched. Of course, it is unlikely that each 
research question will be studied, but I think the research question should start from the work of 
professionals by articulating why and how that can be a guide for professional action. And we 
should be very clear that the research does not provide us with a recipe, but it is professionals 
who decide how external evidence can be integrated into their work, if at all. Rather than being 
subordinate to the notion that they have to implement evidence-based interventions, I think TRIO 
programs should step back and intentionally observe and articulate what is the logic of evidence 
use in TRIO programs. 


But how can we move ourselves to articulate and address important questions? How can we 
articulate our logic? After experiencing programs that required grantees to use “evidence-based” 
intervention, I think the grant-making field is becoming more aware of the limitations of an 
ovetreliance on evidence-based intervention. For example, Wandersman and colleagues (2016) 
report that the 2015-2020 Teen Pregnancy Prevention grant program by the Office of Adolescent 
Health at Health and Human Services “has taken lessons” (p.14) from the previous grant cycle of 
2010-2015 that required applicants to implement evidence-based interventions from its list. The 
new grant cycle still requires grantees to implement evidence-based interventions from the list, 
but it also includes funds to provide capacity-building assistance to organizations on 
implementing evidence-based interventions. I think it is a hopeful sign because it opens up a 
space for a discussion on how to make evidence-based intervention work and how to take up 
local knowledge. I hope the grant-making of TRIO programs will be affected by this policy 
shift. 


Going back to the question of how we can go about articulating our logic and how we can 
generate questions, I list ongoing activities in the TRIO community and resources that could 
contribute to supporting each program. As I discussed above, because the professional’s work 
involves complex decision making, what is important could vary, and it is the professionals who 
have to make intelligent decisions. There is not one size fits all approach for professionals to 
articulate logics for evidence use. So, I list some of the resources and tools that may be helpful. 
Regional meetings and conferences can be a place where TRIO practitioners can compile 
research questions and look for knowledge resources. TRIO program personnel view these 
meetings as helpful and they get ideas for improving their practices. It would be helpful as a 
community to know how this learning process happens, and if there are better ways to promote 
knowledge sharing and transfer. 


e Logic model: As a part of online community of practice activity, TRIO programs have 
shared tools that can be used for better program planning and evaluation. As described 
during the webinar of the COE’s Research, Evaluation, and Data Use Community of 
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Practice (CoP), a logic model is a helpful tool to document theories and gap in knowledge 
(Wahl, April 20, 2016). Logic models could be also helpful for assessing if intervention 
is ready to be researched (Epstein & Klerman,2012). 


e Development of evaluation capacity: Program evaluators have developed tools, 
assessments, and guiding questions that can be used in the program meetings and to 
develop evaluation capacity (for example, Buckley, et al, 2015 on teaching of evaluative 
thinking, which provides strategies and examples of activities for promoting evaluative 
thinking). Action research, as shared in the CoP webinar, can provide contextualized 
knowledge to fill in the gap between what is known from external research and how it is 
rolled out in a program (Arreola, Pena, et al., 2015; Rohse & Kaplan, January 28, 2016). 
Empowerment Evaluation has been used in public health programs and there are tools 
developed to support programs bridging the gap between evidence-based interventions 
and practice. For example, Getting to Outcomes (GTO) provides 10 steps for 
organizations to plan, implement, and evaluate their programs (Rand Corporation, no 
date; Wandersman, et al., 2016) to educational organizations. Both action research and 
empowerment evaluation take the importance of context in which the program unfolds, 
which TRIO practitioners also need to be able to articulate when they develop the logic 
of their work. 


Final Thoughts 


In this essay, I portrayed CPP as a reflection of idiosyncrasy in policymaking, which 
consequently minimized the considerations of how professionals learn and improve practices. I 
think this is a lost opportunity for both the federal government and TRIO programs. Some TRIO 
programs took CPP as a hoop to jump through, which will possibly lead to symbolic adoption of 
a new practice. 


I argued that TRIO programs should draft logics of evidence use that align with their work and 
generate research questions. How to develop organizational capacity to use evidence is an 
important element for program improvement, but policy instruments that TRIO programs came 
across did not highlight it, and it is often told at the end of the program cycle. I am throwing out 
a proposal that this often-invisible process be documented, shared, and discussed, so programs 
can use their logics of evidence use as levers to ask for knowledge that they need for program 
improvement. 
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CHAPTER III 


THE RELATIONSHIP 
BETWEEN RESEARCH AND 
PRACTICE: THE RECENT 
LANDSCAPE 


TRIO, THE WHAT WORKS CLEARINGHOUSE, AND THE 
COMPETITIVE PREFERENCE PRIORITIES (CPPS): AN 


IMPOSED STRUCTURED PRACTITIONER-RESEARCH 
COLLABORATION BY MARGARET CAHALAN 


Abstract 
The “imposed _ structure” of 
incorporating competitive 


preference priorities (CPPs) for the 
use of evidence-based strategies as 
defined by the What Works 
Clearinghouse (WWC) has _ the 
potential to fundamentally redefine 
the relationship between practice 
and research within federal grants in 
education. Given the focus of this 
compilation on practitioner-research 
collaboration in this essay, we 
explore the implications of these new 
regulations and associated 
competitions for TRIO programs and 
service. We present an overview of 
the WWC history and the content of 
the reviews in the areas of relevance 
to TRIO. Then we review the results 
of the specific application in the SSS 
competitions. Finally we discuss the 
limitations of using the WWC and 
give some reflections on making the 
use of “evidence based research” 
more helpful. 


ea: 


Reflecting the articulated commitment of the 
Obama administration to foster the increased use of 
“evidence” based policies to achieve program goals, 
in 2013 the Department of Education regulations 
known as EDGAR were officially modified to allow 
the Secretary of Education to foster increased use of 
“evidence-based” strategies through the awarding of 
competitive financial advantages to programs that 
have “evidence of success” (Federal Register 2013- 
08-13). These modifications provided authority to 
the Secretary to use “absolute priorities” or 
“competitive preference priorities (CPPs)” in grant 
competitions to give priority to those projects that 
were using “evidence-based” practice, and that 
proposed to evaluate these practices using rigorous 
evaluation procedures. The definitions of and 
standards for “research evidence” specified in the 
EDGAR regulations were those adopted and put 
forth by the What Works Clearinghouse (WWC). 


These new EDGAR regulations were consistent 
with the 2012 OMB circular calling for using 
Competitive Preference Priorities (CPPs) in 
competitions for existing programs that are 
“difficult to change.” TRIO became an obvious 
target for applying the new regulations as the largest 
discretionary competitive grant program in the U.S. 
Department of Education which is intentionally 
structured to have a high level of ongoing funding 
of existing programs. Discretion was exercised by 
the Secretary to use the CPPs for the next TRIO 
competitions following the above noted formal 
modification of the EDGAR regulations _ that 
occurred in 2013. The Student Support Services 
(SSS) competition in 2015; the Talent Search 
competition of 2015-2016, and the Upward Bound 
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competition of 2016-17 each utilized CPPs as a method of fostering the implementation of 
evidence-research based strategies. This “imposed structure” of incorporating competitive 
preference priorities (CPPs) for the use of evidence-based strategies as defined by the What 
Works Clearinghouse (WWC) has the potential to fundamentally redefine the relationship 
between practice and research within federal grants in education. Given the focus of this 
compilation on practitioner-research collaboration in this essay, we explore the implications of 
these new regulations and associated competitions for TRIO programs and services. 


Specifically in this essay we: 
1. Give a brief overview of the history and the definitions developed by the WWC; 


2. Present an overview of the Interventions, Single Study Reviews, and Practice Guides in 
the WWC that are within the two content areas of most relevance to TRIO (Pathways to 
Graduation and Postsecondary); 


3. Use SSS as an example and present some preliminary data on the impact of the CPP 
points on the 2015 SSS competition scores; and 


4. Discuss the identified limitations to use of the WWC and give some reflections on 
making the “using evidence based research movement” more useful. 


Overview of the What Works Clearinghouse (WWC) History 
and Growth as the Major Arbiter of Research Legitimacy for 
Federally Funded Education Programs 


The WWC was established in 2002 by ED with a formal mission to “be a resource for informed 
education decision making” by identifying those interventions “that work” so that practitioners 
could have increased guidance to inform practice. The Clearinghouse was also charged with 
producing user-friendly guides for educators on effective instructional practices in order to 
understand what instructional programs have been shown to be effective. The WWC replaced 
the ERIC annotated bibliographies and research summaries that had been developed in the 1980- 
1990s to provide summaries of research in the field. The website asserts that for over a decade, 
the WWC has been “a central and trusted source of scientific evidence for what works in 
education to improve student outcomes.” 


Contract Organization and Costs. The work of the WWC is conducted under a set of 
contracts held by several leading firms with expertise in education and research methodology, 
and managed by the Institute for Education Sciences (IES), the unit in ED focused on research 
and evaluation. Since its inception in 2002, ED has held competitions at roughly five-year 
intervals for the major contract to implement and manage the WWC. Periodic smaller 
competitions for additional work have also been held. The major contractors in 2017 were 
Mathematica Policy Research (MPR), Development Services Group (DRS)—tesponsible for 
postsecondary reviews, Inc. and Sanametrix, Inc. The WWC is not an inexpensive operation. 
The costs of the WWC contracts since 2006 have averaged well over $10 million per year across 
the major contract and sub-contracts. 
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Enshrining the WWC in U.S. Department of Education Regulations. As noted, the 
Notices of Inviting Applicants for the most recent TRIO grant solicitations programs utilize the 
procedures, definitions and ratings developed by the What Works Clearinghouse (WWC 2014). 
Following the general guidance from OMB, ED has taken a tiered approach that has three levels 
that are summarized below. Figure | gives the definitions of Randomized Control Trial (RCTs) 
and Quasi-Experimental Design (QED) needed to meet the Moderate Evidence requirement of 
the CPPs. 


e Promising for developmental studies—Strong theoretical conceptualization and 
correlational evidence 


e Moderate Evidence—Meets WWC methods standards “with or without 
reservations.” This involves interventions that have at least one well-designed 
and implemented Randomized Control Trail (RCT) or at least one well-designed 
and implemented Quasi Experimental Design (QED) that allows establishing a 
positive significant and substantive causal inference between the intervention and 
outcomes and no negative significant relationships. Based on this evidence, 
interventions studied are evaluated as “effective,” thus “evidence-based.” Well- 
executed RCTs can meet WWC standards “without reservations.” Well-executed 
Quasi-experimental designs (QED) can meet WWC standards “with 
reservations.” 


e Strong Evidence—Meets WWC methods standards “without reservations” with 
significant positive outcome and substantive effect size. This involves 
interventions that have at least one well-designed and implemented RCT that 
establishes significant and substantive positive relationships and no negative 
relationships. 


Rationale of the CPPs. While the Trump administration in its FY2018 budget proposals has 
seemingly returned to and extended the Bush Administration’s tactic of selectively using 
research evidence to support its proposals for advancing, cutting or eliminating federal programs, 
the Obama Administration articulated a more systematic approach. This approach fostered 
evidence-based policy making and use of evidence as keys to “make government work 
effectively’ (OMB, May 18, 2012; OMB 2016). The government-wide strategies sought to 
allocate more funds to programs that were supported by stronger evidence, and to require 
rigorous evaluations to produce “strong” evidence. This policy included supporting social 
innovation funds in which the government paid more to programs that could meet required 
outcomes. For existing grant programs, the OMB encouraged agencies to “provide points or 
significant competitive preference priorities to programs that are backed by strong evidence” 
(OMB, May 2012). These strategies were based on the idea that the government should invest 
taxpayer money on programs that are most likely to bring results (Haskins and Margolis, 2014; 
Orszag 2009). 
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Randomized controlled trial means a study that employs random assignment of, for example, 
students, teachers, classrooms, schools, or districts to receive the intervention being evaluated 
(the treatment group) or not to receive the intervention (the control group). The estimated 
effectiveness of the intervention is the difference between the average outcome for the treatment 
group and for the control group. These studies, depending on design and implementation, can 
meet WWC Evidence Standards without reservations. 


Quasi-experimental design study means a study using a design that attempts to approximate an 
experimental design by identifying a comparison group that is similar to the treatment group in 
important respects. These studies, depending on design and implementation, can meet WWC 
Evidence Standards with reservations(they cannot meet WWC Evidence Standards without 
reservations). 


Applications for New Awards; Student Support Services Program A Notice by the 
Education Department on 12/18/2014 ) 


The Movement from Studying Outputs and Outcomes to Using Research Evidence. 
Newcomer (2016) has done historical research tracing the growth of emphasis within 
government agencies from what she calls a stress on measuring “outputs and outcomes” to the 
use of “Demonstrated Evidence-Based Interventions” (DEBIs). She outlines the movement from 
reports such as the Hatry’s (1967) Senate report on Measuring Program Effectiveness, and 
groups such as the World Bank calling for developing outcome measures of the 1990s and the 
Government Performance Results Act (GPRA) of 1994 to the Executive Order of the Bush 
Administration establishing the Program Assessment Rating Tool (PART). Faced with intense 
criticism of the Bush OMB PART order, the Obama Administration discontinued PART and 
subsequently issued new OMB Guidance on Tiers of Evidence that form the basis of the Obama 
Administration’s approach. Newcomer also identifies key challenges faced with regard to the 
use of DEBIs and concludes that there is an overstating of the “ease of the flow of evidence to 
practice.” She cites key challenges, including inadequate attention to support factors and 
understanding the way causal mechanisms need to work together to produce expected results, 
and the focus on replication with fidelity rather than adapting models to changing contexts. She 
also notes that measuring fidelity of implementation and impact are very expensive and often 
difficult ethically. 


The Shift from Asking “Does this Program Work?” to Asking “What Strategies Work 
and Under What Circumstance?” Under the Obama administration there was a change to 
move away from the emphases on evaluating federal programs as a whole which had dominated 
the period of the 1970s to the early 2000s to a move to look at the efficacy of specific strategies 
within a program. Given the broad sweep of federal programs usually intentionally allowing a 
range of services to achieve goals, the overall program evaluations suffered from a lack of clear 
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specification of the actual “treatment” being studied and also a lack of control over the extent to 
which the control group or comparison group was also getting the services due to ethical 
considerations. They tended to be black-box evaluations with numerous study error issues, but 
they were often used to justify reduced or level funding, or in the case of the Upward Bound 
program, using what were later found to be “error filled” conclusions, to justify the “zero 
funding” recommendations of the Bush Administration’s budgets in FY2005 and FY2006 
(Cahalan and Goodwin, 2014). To some extent, this shift to focus on specific strategies offered a 
relief from the defensive position in which most federal programs increasingly found themselves 
at the end of the 20" century due to these overall evaluations. 


Researcher-Practitioner Collaboration--The Logic Flow of the Goals of the CPPs. 
Figure 2 outlines the logic flow of the articulated goals of the use of the CPPs in the TRIO 
competitions held under the Obama Administration. As stated in the “Invitations to Apply”, ED 
articulated the belief that use of evidence-based practices will result in both “a better competition 
and in better results for students.” ED also articulated its interest in possibly partnering in 
research with the grantees in the priority topic. As the logic flow indicates, there is 
anticipated spillover impact toward increasing general evidence-based practice and research use 
throughout the government and in non-profit social programming. The 2015 SSS application 
stated: 


“In recent years, the Department has placed an increasing emphasis 
on promoting evidence-based practices through our grant 
competitions. We believe that encouraging applicants to focus on 
proven strategies can only enhance the quality of our competitions and 
the outcomes of students who participate in our programs........ The 
Department is sufficiently interested in this priority topic that we may 
later seek to partner with successful applicants to conduct research 
and evaluation” (2015 Invitation for Applications for Student Support 
Services Grants). 


Selected Related Research on Research Evidence Use. While the use of evidence-based 
practices within the U.S. Department of Education is too new to have a large body of 
information, scholars have generally been more cautious about the promise of evidence-based 
policymaking than government agencies. Sutherland and colleagues observed that “the 
normative claim that policy should be grounded in an evidence base is itself based on surprising 
weak evidence” (Sutherland et al; 2012, cited by NRC, 2012). A report entitled the Science of 
Using Science from the EPPI Center at the University of London (Langer et al., 2016) presents 
results from a comprehensive meta-analysis of the efficacy of various strategies to increase 
research use by decision makers. 
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The EPPI report found some evidence of positive effects with the following interventions: 


e Those that facilitated active access to research evidence through communication and 


evidence repositories; 


e Those that built on decision-makers’ skills to access and make sense of evidence, and 


e Those that foster changes to the decision making structures, incentives, and processes. 
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Review of WWC Relevant Interventions, Single Studies and 
Practice Guides 


The WWC does not purport to be a source for meta-analysis or systematic review of the 
literature on a given topic such as is the approach of groups such as the EPPI center at the 
University of London, or even in a less systematic manner to take the approach of the ERIC 
Topical Literature Reviews of the 1990s. 


The WWC has evolved over time since its inception, but not in the direction of meta-analyses. 
Initially, there was more of a focus on named “Interventions,” for which there could be a search 
of the literature for research with regard to a given named “intervention.” This approach has a 
number of pitfalls, including the fact that the named “interventions” often were dynamic 
programs that changed over time, and that often few studies could be found that met the 
standards, and these were not necessarily representative of the project at the current time. More 
recently, there seems to be a greater focus on Single Study Reviews and Practice Guides. The 
three types of reports constituting the current searchable WWC data base are: 


e Intervention Reports; 
e Single Study Reviews, and 
e Practice Guides. 


The WWC initially was primarily focused on K-12 topics; however, prompted by the increased 
use of the WWC data base for postsecondary federal competitions, there has been additional 
focus in college access and success topics and strategies since 2012. These are categorized under 
the key words “Pathways to Graduation” and “Postsecondary” in the WWC data base. Figures 
3-6 summarize the results of the WWC reviews for Intervention Studies and for Single Studies in 
these two areas. 


The Category of “Intervention Studies” Reviewed. Figure 3 gives summary information 
and Figure 4 lists the names of the “Interventions” that have been reviewed by the WWC under 
the general categories of Pathways to Graduation, which may involve either high school or 
college graduation; and under the category Postsecondary. Interventions are cross-listed in the 
system so that all of the 7 Interventions listed under Postsecondary are also listed under 
Pathways to Graduation category. As of December 2016, there were 35 named “Interventions” 
that had been reviewed. Typically this means that more than one potential study has been found, 
or that the named program has been around for more time. Among those Interventions without 
any studies meeting standards there are a number of well-known programs. Of the 35 
Interventions that WWC studied in these topic areas, 13, or 37 percent, had no study that met the 
methods standards for the WWC. Those without any studies meeting the methods’ standards 
cannot be reviewed as to effectiveness by the WWC. Notably this listing of programs with “no 
studies” meeting the WWC methods standards includes well-known programs such as “I Have a 
Dream,” “Puente,” “Residential Learning Communities,” “Communities in Schools,” and several 
others (see those coded as 99 in Figure 4). 


ll 


Topical Number of Number of Number with at | Number Percent of 
Classification | interventions in | interventions least 1 study with at least | Interventions 
data base under | listed with no meeting WWC one reviewed with no 
the topic (note studies meeting | methods significant positive results 
there is double WWC methods standards with positive (either no studies 
listing) standards or without result met the methods 
reservations standards or met 
and no impact) 
Pathway to 35 13 22 14 60% 
Graduation 
Postsecondary | 7 1 6 3 57% 
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1. ACT/SAT Test Preparation 
and Coaching Programs 

2. First Year Experience 
Courses for Students in 
Developmental Education 
3. Career Academies 


4. Check & Connect 


5 


6. Reconnecting Youth 
7. Linked Learning 


8. Residential Learning 
9. National Guard Youth 
ChalleNGe Program 


10. Service and Conservation 


11. Communities in Schools 


Name of Intervention 


Year of 
Review 


Methods 
rating: 
99 = none 


standards 
1 = Met 
without 
reservation 
2 = Met 


reservation 
3 = mixed 
of 1 and 2 


Effectiveness 
rating: 

3 =Positive 
2=Potentially 
positive 

1 = Mixed 
0=No 
discernable 
impact 

5 {le 
Potentially 
negative 

-2 =Negative 
99= No studies 
met standards 


Highest 
effect size 
reported 


Pathways to Graduation section (35 Interventions Reviewed) 


. Credit Recovery Programs 


Communities 


Communities 


Corps 


12. Youth Build 


13. High School Puente 
Program 
14. Coca-Cola Valued Youth 
Program 


15. Summer Training and 
Education Program (STEP) 


16. I Have a Dream 


17. Middle College High School 


2016 


2015 


2009 
2009 
2009 
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io’) 


Number of 
studies 
meeting 
methods 
standards of 
those reviewed 


NA 
NA 


NA 


Z| Z| 4 
>| >| > 


Z 
> 


lof 1 


Z 
> 


lofl 


18. Talent Development Middle | 2009 NA 
Grades Program (TDMG) 

19. Wyman Teen Outreach 2009 NA 
Program (TOP) 

21. Accelerated middle schools | 2008 3 of 3 


29. Quantum Opportunity 2007 1 1 of 2 
Program 

30. Talent Development High 2007 1 2 1 of 2 
Schools 


33. Financial Incentives for 2007 1 1 2 of 2 
Teen Parents to Stay in 
School 


Postsecondary category (7 Interventions Reviewed) 
and Coaching Programs 
Courses 


4. First Year Experience 2016 1 1 of 19 
Courses for Students in 
Developmental Education 


6 of 27 


4 of 97 


bridge programs 
a en a a ae 
Communities 
Poms to ke 
Communities 


4 of 16 


The Category of “Single Studies” Reviews. Over time the WWC developed more emphasis 
on what they call” Single Study Reviews.” Of 160 single studies under Pathways to Graduation 
for which methods were rated, there were 36 studies that met the methods standards, and 23 of 
the 160 also had a least one positive result—about 14 percent. Among those categorized as 
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Postsecondary, about 212 were reviewed for methods, and of these, 62 met the methods 
standards. Of the 212 studies, there were 44 that both met the methods standards and had at least 
one positive result (21 percent of total reviewed). Note that Single Studies like the Interventions 
may be doubly classified under Postsecondary and Pathways to Graduation. Considering all 
topics, there are about 2,349 total Single Studies that have had methods reviewed and are 
classified in the WWC data base. 


A review of the single studies (both impactful and not impactful) reveals that many of the 
strategies examined in the Single Study reviews (and also the Intervention reviews) have similar 
conceptual frameworks and seemingly are implementing similar practices. This suggests that 
differential judgements as to whether there is positive impact may be more about the study 
structure, the context, and comparison group-counterfactual differences rather than the 
differences in the strategies. Additional work is needed to assess if there is a pattern of 
differences between those strategies that show positive impact and those that show no impact. 


One also notices that the role of the WWC in the federal grant competitions, especially the TRIO 
postsecondary grants (SSS), Talent Search (TS), Upward Bound (UB) which have required use 
of Single Study reviews, have resulted in an increase in both the number and the percentage of 
studies in the WWC in these areas that have positive impacts. Thus the WWC’s role of doing 
somewhat systematic reviews of an “Intervention” has evolved to focus on Single Studies and 
Practice Guides focusing on those studies with positive results. 


The Category of “Practice Guides.” Since its inception, the WWC has published 22 Practice 
Guides. Of these two are classified under Pathways to Graduation and one under 
Postsecondary. These are: Helping Students Navigate the Path to College: What High Schools 
Can Do, published in 2009; Dropout Prevention published in 2008; and Strategies for 
Postsecondary Students in Developmental Education — A Practice Guide for College and 
University Administrators, Advisors, and Faculty published in 2016. The 2016 guide lists six 
recommendations that range from minimal to moderate in evidence rankings by the WWC. The 
recommendations are as follows: 


1. Use multiple measures to assess postsecondary readiness and place students. 
2 Require or incentivize regular participation in enhanced advising activities 
a Offer students performance-based monetary incentives. 

4. Compress or mainstream developmental education with course redesign. 

3. Teach students how to become self-regulated learners. 

6. Implement comprehensive, integrated, and long-lasting support programs 


5 


Type of Number of | Number RCT Quasi- | At least Atleast | At least | Percent 
Report Single meeting Experi | one one one that met 
Studies methods mental | positive Quasi positive | standard 
with standards with RCT Experi | result and had 
methods or without mental at least 
rated reservations positive one 
positive 
result 
All topics in | 2349 877 (37% with | 576 597 145 87 280 12% 
WwWwCc or without) 
--572 without 
--322 with 
Pathways to | 160 36 (29% with or | 33 23 20 3 23 14% 
graduation without) 
--24 without; 
--15 with; 
Post- 212 62 (29% with or | 38 24 26 17 44 21% 
secondary without) 
--37 without; 
--25 with 


The Special Case of the National Evaluation of Upward Bound. The 2009 Practice 
Guide entitled Helping Students Navigate the Path to College: What High Schools Can Do, 
prepared by Mathematica Policy Research, included a review of the then recently-released 
National Evaluation of Upward Bound final report (Sefter et.al. 2009) and also the Mathematica 
UB evaluation Third Follow up report released in 2004 (Myers et.al. 2004). The review ignored 
the fact that the original and final two technical monitors for the long-running study in the U.S. 
Department of Education had, after a Quality Assurance Review that involved both external and 
internal analysis of all the data files from the study, found that the Mathematica reports in 2004 
and 2009 were seriously flawed in their conclusions that the UB program had no discernible 
impact on postsecondary entrance or degree attainment other than the attainment of vocational 
certificates. In contrast, the analysis done by the Department of Education technical monitors 
had found that Upward Bound had statistically significant and substantial positive impacts on 
postsecondary entrance, financial aid award, and degree attainment (Cahalan and Goodwin 2014; 
Cahalan, 2009). Despite replication of the technical monitors’ findings by Nathan, 2013; and 
Harris, Nathan, Marksteiner, in 2014, the Practice Guide has not been corrected in its reporting 
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concerning the Upward Bound Program. The flawed Mathematica study has had a negative 
impact on Upward Bound program’s reputation and identity as an effective college access 
program. The corresponding undeserved “ineffectual PART rating” led to zero funding requests 
in the FY0O5 and FY06 Bush Administration budget requests. This experience has led to a 
profound mistrust among the TRIO practitioners to contract evaluation research studies that is 
only now beginning to be overcome. 


Using the 2015 Student Support Services (SSS) Competition as 
an Example: Preliminary Data on the Impact of the CPP 
Points in the Competition Scores 


As the data above indicates, there are not a large number of strategies that meet the WWC 
methods standards, have positive results, and also are feasible within the legislatively-defined 
service and funding options of the SSS program. Moreover, those strategies that are applicable 
and have positive impacts often have relatively small effect sizes. For the 2015 SSS 
competition, the Department of Education identified two priority areas. The substantive areas in 
which the Department invited CPP Submissions were: 


e Strategies to Influence the Development of Non-Cognitive Factors Supported by Moderate 
Evidence of Effectiveness (up to 3 points) 


e Providing Individualized Counseling for Personal, Career, and Academic Matters Supported 
by Moderate Evidence of Effectiveness (up to 3 points) 


As part of the SSS competition, the Department provided to applicants citations for three specific 
research articles applicable to the priority areas (See Figure 6) that the WWC had determined 
both met the methods criteria and had at least one positive result. 


e Social belongings intervention: Walton, G. M. & Cohen, G. L. (2011). A brief social-belonging 
intervention improves academic and health outcomes of minority students. Science 331, 1447-1453 
Online Available: http://www.sciencemag.org/content/33 1/6023/1447.full 


e 6©Difference education intervention: Stephens, N M., Hamedani, M G., & Destin (2014). Closing the 
social-class achievement gap: A difference-education intervention improves first-generation students’ 
academic performance and all students’ college transition. Psychological Science 1-11. Online 
Available: http://www.psychology.northwestern.edu/documents/destin-achievement.pdf 


e Individualized Counseling-Inside Track: Bettinger, E. & Baker, R. (2011). The effects of student 
coaching in college: An evaluation of a randomized experiment in student mentoring. Online 
Available: https://ed.stanford.edu/sites/default/files/bettinger_baker_030711.pdf 
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One of the two studies in the non-cognitive topic area was an experimental intervention that took 
about one hour to implement (Walton, G. M. & Cohen, G. L., 2011). The small sample from a 
selective university was 37 Blacks and 33 Whites. Modest positive impacts were observed for 
Blacks but not Whites when compared to a comparison group, which was given a one-hour 
session that did not involve the social belonging element. The other study in the “non-cognitive” 
priority area was an orientation experiment for first-generation students that involved about three 
hours (Stephens, N M., Hamedani, M G., & Destin, 2014). 


The one study citation given by ED for the individual counseling priority area involved a 
randomized control trial with 3,527 students for an intervention of the use of a commercial on- 
line software tool for college coaching and mentoring —Inside Track (Bettinger, E. & Baker, R. 
(2011). This research found a 4 percentage point difference in 12-month persistence, but no 
differences in completion of a degree in 4 years. One might expect that in the case of SSS, the 
expected impact would depend on how different this was from what the projects were already 
doing. 


SSS applicants were free to find and submit other articles within the priority areas that met the 
WWC criteria and had at least one positive result; however, out of some 1,500 applicants, only 
17 applicants identified another intervention study that met the criteria. The competition for 
TRIO grants is very intense, and in order to be competitive, it is necessary to successfully 
address the competitive priorities. The data displayed in Exhibit 5 confirms that 95 percent of 
the applicants chose to address the CPPs and over three-fourths (77 percent) were able to obtain 
the full 6 points, which means that they proposed to implement the specific research-based 
programming in both of the areas (non-cognitive and student coaching). A review of their status 
as “new” or “previously funded” applicants indicates that most of the applicants who did not 
choose to address the CPPs were new applicants. Given SSS scoring, it was not possible for new 
applicants to win unless they addressed the CPPs. The cut-off scores each cycle depend on the 
amount of funding available relative to the number of applicants. For 2015, there were 1,480 
applicants and 1,071 awards. The cut-off score was 104, meaning that a new applicant would 
need to get a perfect or near perfect score on the Project Design section (worth 100 points) and 
also obtain 4 of the 6 CPP points. In the TRIO competitions held every five years, existing 
projects may earn up to 15 extra points for meeting their objectives specified in the previous 
cycle years. These points are called Prior Experience (PE) and are tabulated by an independent 
contractor based on submitted performance reports. These points give existing projects an edge, 
and there is a high degree of project continuance. The experience of past competitions funding 
cycles, in which about 2/3 of applicants are funded and 1/3 are not funded, led COE to advise 
applicants to try for the CPP points even if they believed they would get most or all of the Prior 
Experience (PE) points for the cycle. 
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Discussion of the Identified Limitations to the Use of the 
WWC and Reflections on Making the “Evidence Based 
Practice” Movement” more Useful 


As discussed, the Department of Education has chosen the WWC Standards as the way to 
provide an authoritative independent judgment concerning what constitutes “research evidence” 
for implementation of its CPPs. The use of WWC contractor ratings has the advantage of 
removing ED from technical decision making in this regard, and in having published standards 
available to all. However, throughout its history, since the early 2000s, IES and the WWC have 
not been without criticism, and these limitations can be expected to influence the efforts at 
implementation of the increased evidence use in the federal education programs. 


Below is a summary of some of the critiques of the WWC, followed by some ideas moving 
forward that might lead to a more meaningful use of research to foster program improvement. 


e Favoring Research Capability over Need for Services. One of the criticisms of the 
CPPs for 2015, in the public comment period, was that the CPPs would promote undue 
preference to research institutions and put smaller, less resourced, rural, and minority-serving 
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institutions at a disadvantage in the grant-writing competition process. Over time, the larger 
IHEs with grant-writing staff increasingly submit grants written by professional grant writers 
other than the TRIO staff at the institution. For example, several of the larger IHE’s now 
have multiple SSS grants--(Regular, STEM, Disabled, Veterans). Commenters questioned 
whether the Department was not putting “Research Capability” over the “Need for Services.” 
Additional research is needed to observe if this was indeed the case. 


e Narrowness of WWC Compared to Wider Conceptualizations of Evidence to the 
Exclusion of Those Practices that Might be of Most Benefit. Another criticism of 
the WWC as the arbiter of which practices are to be promoted concerns the narrowness of the 
standards for what constitutes “evidence.” The methods favored by WWC ignore a wide 
body of non-experimental observational science. Those of us who have attempted to 
implement ethical random assignment studies in the time-sensitive area of college access 
know that it is only feasible to implement a very narrowly-focused random assignment in 
which almost equally resourced interventions are offered and then it becomes very difficult 
to assess impact. 


We can distinguish three concepts relating to research evidence of which the WWC 
definition is the narrowest: 


1. WWC Standards Based Research Evidence; 
2. Science-Based Research Evidence, and 
3. Evidence-Based Practice (EBP). 


Figure 8 illustrates our conceptualization of these three types of research based evidence in terms 
of the narrowness of the concepts. Evidence-Based Practice (EBP) is an encompassing concept 
that takes into account the “built up clinical knowledge,” client interactions, implementation 
findings, as well as Science-Based Research Evidence (Soyden and Palinkas 2014). Science- 
Based Research is defined more broadly than the WWC ---as research activity that employs 
systematic, empirical methods to address a specific question. The WWC Standards Based 
Definition is the most narrowly defined of the three. As the tables above demonstrate, the WWC 
Standards Based determination of research evidence is comprised of far fewer studies. By the 
end of 2016, there were about 44 single studies in the WWC that were in the area of 
Postsecondary-- and that met the WWC Standards ---and that had at least one statistically 
significant positive finding. 
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] Evidence-Based Practice (EBP) 


Science-Based Research Evidence 
| What Works Clearinghouse Research Evidence 


Questions about Utility and Validity of Research Rankings of WWC. In the early 
days of the WWC most of the focus was on K-12 rather than postsecondary education. 
Hence, most of the critiques have addressed the ratings of the studies and conclusions with 
regard to the interventions at the K-12 level. Alan Ginsberg and Marshall Smith (2016) 
conducted a review of 27 RCT studies of math curricula in the WWC, and found that 26 of 
the 27 had serious threats to validity or usefulness other than that of the selection bias that the 
RCT method controls. Moreover, they note that the magnitude of the error generated by the 
threats was often greater than the average effect size of the RCT treatment. They 
recommended a panel of experts and users be established to consider how to improve the 
WWC criteria and standards for review. 


High Level of Changes in Ratings. Other critiques have noted the high rate of changes 
in WWC ratings following a request for a re-review of the findings, usually by study authors. 
For example, the National Institute for Direct Instruction (NIFDI) filed a Freedom of 
Information Act request for data on the results of the Requests for Reconsideration made to 
the WWC. NIFDI found that of the 69 requests reviewed 54 resulted in changes in ratings. 
The NIFDI reviewers also pointed to the high rate of exclusion of research studies as not 
meeting the WWC criteria. NIFDI has been among the most severe of the critiques of the 
WWC, especially related to the WWCs conclusions with regard to their highly studied 
reading curriculum. Siegfried Engelmann, founder of Direct Instruction, notes: 
“Unfortunately, the WWC has failed to live up to its promise. The WWC's reports promote 
curricula that the scientific community has found to be ineffective and inefficient and 
denigrate those that the scientific community has found to be highly effective.” 
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Englemann concluded that the WWC was so “irreparably biased that it would have to be 
thoroughly reoriented and reorganized under different management rules to perform the function 
of providing reliable, accurate information about what works.” 
http://www.nifdi.org/research/reviews-of-di/what-works-clearinghouse. 


After filing the FOIA request from the WWC, NIFDI researcher Wood (2014) summarized the 
issues: 


With the information provided from the FOIA request and the publicly 
available information, three conclusions appear clear: 1) The WWC 
suffers from a lack of transparency in their policies and guidelines, 2) 
the conclusions they create in their reports can be misleading, and 3) 
the reports are potentially damaging to program developers and 
ultimately the success of students (Wood, 2014). 


e Small Percentage of Studies Meeting WWC Methods Criteria and Low Effect 
Sizes. A recent review (Malouf D. and Taymans 2016) of the total WWC “Interventions” 
reviewed points out (as have others) the small percentage of studies meeting the methods 
criteria. They also noted the small effect sizes of those studies that were listed as meeting 
criteria and having positive results. The authors concluded: 


Most interventions were found to have little or no support from 
technically adequate research studies, and intervention effect sizes 
were of questionable magnitude to meet education policy goals. These 
findings painted a dim picture of evidence based on education 
interventions and indicated the need for new approaches including a 
reexaminiation of federal reliance on experimental impact research as 
the basis for gauging intervention effectiveness (Malouf D. and 
Taymans, 2016). 


e Lack of Concern for the Differences in “Treatment Contrast.” Other researchers, 
outside and inside the context of WWC criticism, have noted the importance of the 
“treatment contrast” and point out that the differential results of RCTs may often have more 
to do with how the study is structured and different counterfactuals than with the differences 
in the interventions. These critiques stress the importance of measurement of both the 
treatment and control groups’ level of receipt of the intervention or its equivalents (Weiss, 
Brock, and Bloom, 2013). The “business as usual” practices for comparison may also 
include substantial practices that are similar to the interventions being studied (Heckman, 
2002). To understand the treatment contrast a researcher must examine the intervention 
related experiences of both treatment and control group members. 


e Unwise Linking of Funding Decisions to Research Findings. Political rhetoric and 
policy of the last 40 years has tended to support the linking of funding decisions to evaluation 
outcomes. This is a dangerous policy that risks failure to serve the most needy and 
vulnerable persons, and also mediates against honest and fair evaluations that lead to 
improved services. As Haskins and Margolis (2014) point out: 
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In our view, an important part of a comprehensive, evidence-based 
strategy will be continuing the funding of programs with initially 
discouraging evaluations. Part of the federal evidence-based culture 
should be that federal agencies will work with programs, and continue 
their funding, as long as they are using evidence to improve their 
outcomes and are showing some progress (Haskins & Margolis, 2014). 


e Promotion of Formulaic Implementation of Handful of Superficial, 
Standardized One Size Fits All Strategies. As the review of single studies that have 
both met the WWC standards requirements and also have achieved some statistical 
significance revealed, there is a limited range of strategies that can be studied with an RCT 
design that does not involve IRB-prohibited denial of services. The tendency for most TRIO 
programs to attempt to replicate the same studies can lead to a less creative response to 
changing situations and co-learning that takes into account the differing situations and needs 
of students. There is insufficient attention to what works and for whom and under what 
circumstances (Imai and Ratkovic, 2013; Weis, Bloom, and Brock, 2013; Gamoran 2014). 


e Demonstrating the Need for Research over Service Needs. Related to the above 
comments, practitioners fear that their service programs will be turned into permanent pilot 
demonstration grants, and that those sites that are not able to do this or do not wish to do this, 
will gradually be competed out of service. Researchers like to cite lack of positive findings 
as a justification for increased need for more research projects and may have a house bias 
towards study designs unlikely to produce positive outcomes. Researchers such as Jon 
Barron of the Coalition for Evidence Policy states that: “The better the evaluations, the more 
likely they are to show that some programs do not produce significant impacts.” Baron goes 
on to report that, out of 90 interventions evaluated by randomized control trials (RCTs) paid 
for by IES since 2002, 88 percent were found to have weak or no positive effects. Similar 
results are produced by RCTs of clinical interventions in medicine (Coalition for Evidence- 
Based Policy). 


e Danger of Favoring of One Intervention over Another with Sparse Evidence. 
There are serious issues in favoring one type of intervention based on a single study that does 
not usually make precise comparisons across strategies or programs. The counterfactual is 
typically simply “business as usual,” which can mean a variety of unmeasured contrasts. 
There could also be potential conflicts with the authorizing legislation for the federal 
programs which allow a wide range of services. Potentially an unfair advantage is given to 
those projects for which the “favored” intervention is appropriate or even feasible. 


Ideas Moving Forward 


Given these criticisms, the following are six ideas concerning use of evidence moving forward. 


1. Decouple Funding Decisions From Results of the Evaluations. This coupling 
discourages innovation and thoughtful implementation and mediates against on-going 
learning from evaluations. 
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2. Address the Serious Validity and Reliability Concerns That Have Been Raised 
Concerning Validity of WWC Results. The WWC provides a formulaic way to 
address OMB requirements, because it seemingly is clear and definitive, but like the 
standardized tests, it is unclear that its overall impact is positive in terms of best practice 
in education. 


3. Align the WWC Standards with Other National Standards, such as those of the 
Joint Committee on Education Program Evaluation Standards; NCES Statistical 
Standards, and AERA Standards for Research Publications. 


4. Broaden the Definition of Research Evidence to include Evidence-Based Practice 
(EBP) and Science-Based Research Evidence and invite thoughtful consideration of how 
best to serve students. Consider meta-analyses and weight of evidence, more nuanced 
approaches. 


5. Consider the Implications for Students if Most Projects Implement the Same 
“Cookie Cutter” Strategies regardless of context and the range of services available at 
the institutions. Consider the students’ real needs and the program’s own “niche” in the 
institution or the community. 


6. Consider New Evaluation Methods Such As Empowerment, Participatory and 
Collaborative Evaluation (Federman et. al. 1996; Chinman et.al, 2004; Chinman 
et.al. 2008, CDC Cox et.al., 2010). Consider methods that stress a deeper understanding 
of context and researcher-practitioner-client interactions. As A. Gamoran ( 2014) notes: 


To date, many rigorous studies treat programs as if they were black 
boxes, seeking a positive or negative judgment without aiming to 
understand how the outcomes are reached. The next generation of 
policy research in education will advance if it offers more evidence on 
mechanisms so that the key elements of programs can be supported, 
and the key problems in programs that fail to reach their goals can be 
repaired (Gamoran, 2014). 


REFERENCES 


Ahmad, A.R., Adi, M., Nazir, M., Md Noor, H., Ng, K.S., & Azizan, A.Z. (2013). An assessment 
of agency theory in the context of higher education institutions. 


Bailey, T., Bashford, J., Boatman, A., Squires, J., Weiss, M., Doyle, W., Valen-tine, J.C., LaSota, 
R., Polanin, J. R., Spinney, E., Wilson, W., Yeide, M., & Young, S.H. (2016). Strategies 
for postsecondary students in developmental education — A practice guide for college and 
university administrators, advisors, and faculty. Washington, DC: Institute of Education 
Sciences, What Works Clearinghouse. 


124 


Bettinger, E.P., & Baker, R. (2011). The effects of student coaching in college: An evaluation of 
a randomized — experiment in student — mentoring. Retrieved from 
https://cepa.stanford.edu/sites/default/files/bettinger_baker_030711.pdf 


Bourdieu, P. (1977). Outline of a Theory of Practice. Cambridge University Press: London. 
Bourdieu, P. (1990). The Logic of Practice. Polity Press: Cambridge. 


Bourdieu, P., & Wacquant, L.J.D. (1992). An Invitation to Reflexive Sociology. University of 
Chicago Press: Chicago. 


Cahalan, M. (2009). Addressing Study Error in the Random Assignment National Evaluation of 
Upward Bound: Do the Conclusions Change? Washington, D.C.: The Pell Institute for 
the Study of Opportunity in Education. Retrieved from 
http://www. pellinstitute.org/publications-Do_the_Conclusions_ Change_2009.shtml. 


Cahalan, M., & Goodwin, D. (2014). Setting the Record Straight: Strong Positive Impacts 
Found from the National Evaluation of Upward Bound, Re-Analysis Documents 
Significant Positive Impacts Masked by Errors in Flawed Contractor Reports. 
Washington, D.C. : The Pell Institute for the Study of Opportunity in Higher Education. 
Retrieved from http://www. pellinstitute.org/downloads/ publications 
Setting_the_Record_Straight_June_2014.pdf 


Chinman, M., Hunter, S.B., Ebener, P., Paddock, S., Stillman, L., Imm, P., & Wandersman, A. 
(2008). The getting to outcomes demonstration and evaluation: An illustration of the 
prevention support system. American Journal of Community Psychology. 41: 306-224. 
doi: 10.1007/s10464-008-9 163-2 


Chinman, M., Imm, P., & Wandersman, A. (2004). Getting To Outcomes: Promoting 
Accountability Through Methods and Tools for Planning, Implementation and 
Evaluation. Santa Monica, CA: RAND Corporation. 


Cox, P.J., Keener, D., Woodard, T., & Wandersman, A. (2010). Evaluation for Improvement: A 
seven step empowerment evaluation approach for violence. Center for Disease Control 
(CDC). Retrieved from 
https://www.cdc.gov/violenceprevention/pdf/evaluation_improvement-a.pdf 


Crosse, S., Williams, B., Hagen, C.A., Harmon, M., Ristow, L., DiGaetano, R., Broene, P., 
Alexander, D., Tseng, M., & Derzon, J.H. (2011). Prevalence and implementation fidelity 
of research-based prevention programs in public schools: Final report. U.S. Department 
of Education. Retrieved from https://www?.ed.gov/rschstat/eval/other/research-based- 
prevention.pdf 


125 


Department of Education: Applications for new awards; Student Support Services Program,” 79 
Federal Register, 243, (December 18, 2014), p. 75723. Retrieved from 
http://www.gpo.gov/fdsys/pkg/FR-2014-12-18/pdf/2014-29769.pdf 


Donaldson, S., Christie, C., & Mark, M. (Eds.). (2015). Credible and Actionable Evidence, 2nd 
Edition. Thousand Oaks: Sage Publications. 


Eisenhardt, K.M. (1989). Agency theory: An assessment and review. Academy of management 
review, 14(1), 57-74. 


Eisenhardt, K.M. (1989). Building theories from case study research. The Academy of 
Management Review; Oct 1989; 14, 4; ABI/INFORM Global 


Epstein, D., & Klerman, J.A. (2013). When is a program ready for rigorous impact evaluation? 
The role of falsifiable logic model. Evaluation Review, 36 (5), 375-401. 


Engelmann, S. (2013). Machinations of What Works Clearinghouse. Retrieved from 
http://eppi.ioe.ac.uk/ems/ The Evidence for Policy and Practice Information and Co- 
ordinating Centre (EPPI-Centre) is part of the Social Science Research Unit at the UCL 
Institute of Education. 


Fetterman, D., Kaftarian, S.J.. & Wandersman, A. (1996). Empowerment Evaluation: 
Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage 
Publications. 


Ginsberg, A., & Smith, M. (2016). Do Randomized Controlled Trials meet the “Gold 
Standard”? A study of the usefulness of RCTs in the What Works Clearinghouse. 
Washington, D.C.: American Enterprise Institute. 


Gamoran, A. (2013). Educational inequality in the wake of No Child Left Behind. William T. 
Grant Foundation Spencer Foundation Award Lecture to the Association for Public 
Policy and Management, November 7, 2013. 


Hallfors, D., & Godette, D. (2002). Will the “Principles of Effectiveness” improve prevention 
practice? Early findings from a diffusion study. Health Education Research: Theory & 
Practice, 17 (4), 461-470. 


Haskins, R., & Margolis, G. (2014). Obama’s promise: Using evidence to fight the nation’s 
social problems. Finnigan, K S. & Daly, A J. (Eds). Using research evidence in 
education: From the schoolhouse door to Capitol Hill. New York: Springer. 117-132 


Hatry, H. (1967). Report of the Senate Committee on Measures of Effectiveness for Social 
Programs. Retrieved from 
https://www.jstor.org/stable/244890?seq=1#page_scan_tab_contents 


126 


Heckman, J., Hohmann, N., Smith, J., & Khoo, M. (2000). Substitution and dropout bias in 
social experiments: A study of an influential social experiment. The Quarterly Journal of 
Economics, May 2000. 


Honing, M.I. (n.d.). Complexity and policy implementation: Challenges and opportunities for the 
field. Honing, M. I. (Ed). New directions in education policy implementation: 
Confronting complexity. pp. 1-24. Albany: State University of New York Press. 


Kivist6, J. (2007). Agency theory as a framework for the government-university relationship: 
Tampere University Press. 


Kivist6, J. (2008). An assessment of agency theory as a framework for the government— 
university relationship. Journal of Higher Education Policy and Management, 30(4), 
339-350. 


Laura and John Arnold Foundation. (2016). Demonstrating the power of evidence-based 
programs to “move the needle” on major U.S. social programs: A competition & funding 
opportunity. Retrieved from http://www.arnoldfoundation.org/rfp-demonstrating-power- 
evidence-based-programs-move-needle-major-u-s-social-problems/ 


Langer, L., Tripney, J., & Gough, D. (2016). The Science of Using Science: Researching the Use 
of Research Evidence in Decision-Making. London: EPPI-Centre, Social Science 
Research Unit,UCL Institute of Education, University College London. ISBN: 978-1- 
907345-88-3 


Liefner, I. (2003). Funding, resource allocation, and performance in higher education systems. 
Higher Education, 46(4), 469-489 


Leake, M., & Lesik, S. (2007). Do remedial English programs impact first-year success in 
college? An illustration of the regression-discontinuity design. International Journal of 
Research and Method in Education, 30(1), 89-99. 


Lesik, S. (2011). New Perspectives in Applying the Regression-Discontinuity Design for 


Program Evaluation: A Simulation Analysis. Journal of Modern Applied Statistical 
Methods, 10(1), 198-206. 


Lesik, S. (2007). Do developmental mathematics programs have a causal impact on student 
retention? An application of discrete-time survival and regression-discontinuity analysis. 
Research in Higher Education, 48(5), 583-608. 


Mayne, J. (2010). Building an evaluative culture: The key to effective evaluation and results 
management. Canadian Journal of Program Evaluation, 24(2), 1-30. 


Luyten, H. (2006). An empirical assessment of the absolute effect of schooling: regression 


127 


discontinuity applied to TIMSS-95. Oxford Review of Education, 32(3), 397-429. 


Metz, A. J.R., Blasé, K., & Bowie, L. (2007). Implementing evidence-based practices: Six 
drivers of success. Retrieved from http://www.childtrends.org/wp- 
content/uploads/2013/07/2007-29EVPSuccess.pdf 


Myers, D., Olsen, R., Seftor, N., Young, J., & Tuttle, C. (2004). The impacts of regular Upward 
Bound: Results from the third follow-up data collection. Report submitted to the U.S. 
Department of Education. Washington, DC: Mathematica Policy Research, Inc. 


Malouf, D., & Taymans, J. (2016). Anatomy of an evidence base. Educational Researcher, 
45(8), 454-459 


National Center for Research in Policy and Practice. (2016). Survey of practitioners’ use of 
research. National Research Council. (2012). Using Science as Evidence in Public Policy. 
Committee on the Use of Social Science Knowledge in Public Policy, K. Prewitt, T.A. 
Schwandt, and M.L. Straf, Editors. 


Notice of Final Supplemental Priorities and Definitions for Discretionary Grant Programs, 
published in the Federal Register on December 10, 2014 (79 FR 73426) and from 34 CFR 
77.1. 34 CFR 77.1 


Newcomer, K., & Brass, C. (2016). Forging a strategic and comprehensive approach to 
evaluation within public and nonprofit organizations: integrating measurement and 
analytics within evaluation. American Journal of Evaluation, 37 (1), 80-99. 


Newcomer, K. (2016). Evidence-based interventions and programs: What does this mean for us? 
Presentation to Council for Opportunity in Education Annual Conference, San Diego 
California. 


National Center for Research in Policy and Practice. (2016). Survey of Practitioners’ Use of 
Research. Retrieved from http://ncrpp.org/assets/documents/NCRPP_SPUR- 
Instrument.pdf 


Olejniczak, K., Raimondo, E., & Kupiec, T. (2016). Evaluation units as knowledge brokers: 
testing and calibrating an innovative framework. Evaluation, 22 (2), 168-189. 


Orszag, P. (2009). Building rigorous evidence to drive policy. Retrieved from 
https://www.whitehouse.gov/omb/blog/09/06/08/BuildingRigorousEvidencetoDrivePolic 
y/ 

Office of Management and Budget. (2016). Building and using evidence to improve results: 
Meeting our greatest Challenges: The President’s Fiscal year 2017 budget. Retrieved 
from 


128 


https://www.whitehouse. gov/sites/default/files/omb/budget/fy2017/assets/fact_sheets/Bui 
Iding%20and%20Using %20Evidence.pdf 


Office of Management and Budget. (May 18, 2012). Memorandum on the heads of executive 
departments and agencies. Subject: Use of Evidence and Evaluation in the 2014 Budget. 
Retrieved from https://www.whitehouse.gov/sites/default/files/omb/memoranda/2012/m- 
12-14. pdf 


Office of Management and Budget .(July 26, 2013). Memorandum to the heads of departments 
and agencies. Next Steps in the Evidence and Innovation Agenda. Retrieved from 
https://www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-17.pdf 


Palinkas, L., Garcia, A.R., Aarons, G.A., Finno-Velasquez, M., Holloway, I.W., Mackie, T.I., & 
Chamberlain, P. (2014). Measuring Use of Research Evidence: The Structured Interview 
for Evidence Use, Research on Social Work Practice. Thousand Oaks: Sage Publications. 


Palinkas, L.A., Garcia, A.R., Aarons, G.A., Finno-Velasquez, M., Holloway, I.A., Mackie, T.I., 
Leslie, L.K., & Chamberlain, P. (2015). Measuring use of research evidence: The 
structured interview for evidence use. Research on Social Work Practice, 1-15. 


Ross, S. (1973). The economic theory of agency: the principals’ problem. American Economic 
Review, 63, 134-130. Results for America. (2015). Federal evidence-based innovation 
programs. Retrieved from 
http://results4america.org/wpcontent/uploads/2015/10/Innovation-fact-sheet.pdf 


Seftor, N.S., Arif, M., & Schrim, A. (2009). The Impacts of regular Upward Bound on 
postsecondary outcomes 7-9 Years after scheduled high school graduation. Report 
submitted to the U.S. Department of Education. Washington, DC: Mathematica Policy 
Research, Inc. 


Seastrom, M. NCES Statistical Standards (NCES 2003-601). U.S. Department of Education, 
National Center for Education Statistics. Washington, DC: U.S. Government Printing 
Office, 2002. 


Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental 
designs for generalized causal inference. Cengage Learning. 


Stephens, N.M., Hamedani, M.G., & Destin, M. (2014). Closing the social-class achievement 
gap: A difference-education intervention improves first-generation students' academic 
performance and all students' college transition. Psychological Science. Retrieved from 
http://www.psychology.northwestern.edu/documents/destin-achievement.pdf 


Stockard, J., & Wood, T.W. (2013). Does the What Works Clearinghouse Work? A NIFDI 


White Paper National Institute for Direct Instruction and University of Oregon and 
National Institute for Direct Instruction. 


129 


Soydan, H., & Palinkas, L.A. (2014). Evidence-Based Practice in Social Work: Development of 
a New Professional Culture. New York, NY: Routledge. 


Supplee, L.H., & Metz, A. (2015). Opportunities and challenges in evidence-based social policy. 
Social Policy Report 28(4). Retrieved from 
http://www.srcd.org/sites/default/files/documents/spr_28_4.pdf 


Sutherland, W.J., Bellingan, L., Bellingham, J.R., Blackstock, J.J., Bloomfield, R.M., Bravo, 
M., et al. (2012). A collaboratively-derived science-policy research agenda. PLoS ONE 
7(3): €31824. doi:10.1371/journal.pone.0031824. 


Tseng, V. (2013). Studying the use of research evidence in policy and practice. William T Grant 
Foundation. Retrieved from 
http://wtgrantfoundation.org/library/uploads/2015/09/Studying-the-Use-of-Research- 
Evidence-in-Policy-and-Practice.pdf 


U.S. Department of Education. (2016). U.S. Department of Education FY 2015 Annual 
Performance Report and FY 2017 Annual Performance Plan. Retrieved from 
http://www?2.ed.gov/about/reports/annual/2017plan/2015-2017-apr-app.pdf 


Walton, G.M., & Cohen, G.L. (2011). A brief social-belonging intervention improves academic 
and health outcomes of minority’ students. Science, 331, 1447-1451. 
http://web.stanford.edu/&sim;gwalton/home/Publications_files/Walton_Cohen_2011_Sci 
ence_1.pdf 


What Works Clearinghouse (WWC). (2014). Procedures and Standards Handbook Version 3.0, 
March 2014. 
http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_O_standards_h 
andbook.pdf 


Weiss, M.J., Howard, S.B., & Brock, T. (2013). A conceptual framework for studying the 
sources of variation in program effects. MDRC Working Papers on Research 
Methodology 


Wood, T.W. (2014). Examining the inaccuracies and mystifying policies and standards of the 
What Works Clearinghouse: Findings from a Freedom of Information Act Request. 
Technical Report 2014-15. National Institute for Direct Instruction. Retrieved from 
http://www. nifdi.org/docman/research/technical-reports/1241-2014-5-examining-the- 
inaccuracies-and-mystifying-policies-and-standards-of-the-what-works-clearinghouse- 
findings-from-a-foia-request/file 


130 


About the Author: 


Dr. Margaret Cahalan is the Vice President for Research and Director of the Pell Institute for the 
Study of Opportunity in Higher Education, of the Council for Opportunity in Education (COE). Over 
a 30 year career she has directed numerous large sample surveys and evaluation studies. After 
working at Westat, Mathematica Policy Research and RTI, she joined the U.S. Department of 


Education from 2004 to 2011. In this role she served as the Leader for the Secondary, Postsecondary 
and Cross Cutting Division of the Policy and Program Studies Services (PPSS). 


Contact Information: MARGARET CAHALAN, DIRECTOR VICE PRESIDENT FOR RESEARCH, 
The Pell Institute for the Study of Opportunity in Higher Education, 
margaret.cahalan @ pellinstitute.org 


Bl 


RESULTS OF THE ASHE-PELL COLLABORATION’S 


EXPLORATORY LANDSCAPE SURVEY sy mika vamasuita 


The federal TRIO Programs are one of the oldest groups 
of higher education social programs, which originated 
from the Economic Opportunity Act of 1964 and the 
Higher Education Act of 1965 in response to the Civil 
Rights movement and the Johnson Administration’s War 
on Poverty. The programs have infrequently had national 
evaluations focused on the effectiveness of the program as 
a whole. However, there is little systematic 
documentation of what individual programs may be doing 
to evaluate their programs and how they may or may not 
be using research for program development. Although an 
evaluation plan is required in the grant proposal, grantees 
are not required to submit their evaluation reports. For 


Abstract 


This paper — summarizes 
results from a survey called 
the ASHE-Pell Institute 
Collaboration Landscape 
Survey. The purpose of this 
survey was to acquire 
descriptive information about 


how TRIO directors had used 


grantees, submission of an annual progress report (APR) 
is required by the Department of Education. In addition, 
there are no funds specifically allocated for evaluations. 
While we know that TRIO practitioners regularly 
exchange ideas and best practices during an annual 
conference, regional conferences, and other occasions, we 
do not know how TRIO practitioners generate knowledge 
and refine their programs or how they use research and 
evaluation in this process. 


research and evaluation to 
inform their program 
planning, how their program 
had conducted evaluation, 
and what effective strategies 
and practices their programs 
used. 


The Landscape Survey addresses this gap in the 
knowledge in the TRIO community, i.e., how do TRIO 
programs use research and evaluation, how do they conduct their own program evaluation, and 
what is the landscape of TRIO practitioner knowledge? A principal goal of the survey was to 
collect information that can promote future discussions about how to better promote research and 
evaluation use among TRIO programs. In this report, after presenting the survey and response 
rate, observations, and recommendations for further research are presented. 


The Survey, Response Rate, and Respondent Characteristics 


Survey Development. Based on the consensus built during the first Collaboration meeting in 
September 2014 that the ASHE-Pell Institute Collaboration would conduct an exploratory survey 
on TRIO programs to address these questions, a team of ASHE and Pell Collaboration members 
generated core questions and drafted question items between April and July 2015. The survey 
was to collect descriptive information on the following topics: 
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1. Where do TRIO directors seek information to inform their practice and how helpful are 
these sources? 

2. Do TRIO directors use research and evaluation to plan their program? If so, what types 
of research do they use and how do they use it? 

3. What challenges do TRIO directors encounter to use research and evaluation? 

4. Do TRIO programs conduct evaluations of their own program? What types of evaluation 
do they conduct? Who conducts the evaluation? What data do they use for evaluation? 
How do they use the evaluation? 

5. Do TRIO programs collaborate with other organizations or units to do evaluation? 

What questions do TRIO directors want to ask that would help to improve their program? 

7. What is the landscape of practitioner knowledge? What are effective strategies or 
practices that TRIO programs are using? 


ON 


The survey included 80 question items, of which 11 were open-ended questions. 


Survey Administration and Response Rate. The Pell Institute distributed the online survey 
to a total of 1,427 TRIO directors who are members of the Council for Opportunity in Education 
(COE). COE is a membership organization of which TRIO programs are a majority of its 
membership pool.' On July 28, 2015, Pell Institute staff sent an e-mail that described the 
purpose of the survey and encouraged the TRIO directors to fill out the online survey. The link 
to the survey was included in the e-mail. The first page of the survey was a consent form, and 
only consenting respondents could proceed to the survey questions. 


The Pell Institute staff sent two follow-up e-mails to encourage the directors to fill out the 
survey. A total of 80 respondents answered the survey by August 28, 2015, the due date of the 
survey. Given the short time frame of the survey, and the time frame of August when many 
directors are not working and the limited follow up conducted, the response rate was 5 percent. 
Given this response rate, these figures discussed below should be taken as “exploratory results.” 


Respondent Characteristics. Of the 80 respondents, 40 (50 percent) were Student Support 
Services (SSS) directors. Among respondents, SSS programs had higher representation than the 
actual proportion, as in FY 2014, 37 percent of all TRIO programs were SSS programs. The 
high representation from SSS program directors in the respondent group may derive from the 
fact that the survey was conducted during the summer when SSS grant application results were 
announced and SSS directors might have been working, while directors of other programs might 
have been out of the office in August. In addition, the Competitive Preference Priority (CPP) in 
the 2015 SSS competition that required the use of studies meeting the What Works 
Clearinghouse’s (WWC) moderate level of evidence of effectiveness standards almost certainly 
impacted the responses of the SSS directors compared to other respondents. SSS program 
directors’ responses, particularly the use of the WWC, might have been over represented. Given 
this potential bias of the respondent group, in our analysis, we examined whether SSS program 


' Taken together, there are about 2779 TRIO programs (Upward Bound, Upward Bound Math Science, Veterans 
Upward Bound, Student Support Services, Talent Search, Educational Opportunity Centers, McNair). As many 
institutions have more than one of the programs, the 1427 COE memberships represents more than half of the TRIO 
institutions. 
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directors’ responses differed from other program directors. We report the differences in the 
findings section. 


Exploratory Survey Findings 


Below, we present findings on the sources of information, ratings of helpfulness, the extent of 
research use, and the challenges of using research and evaluations in their own programs. 


Information Sources and Usefulness of These Data Sources. The survey responses 
suggest that TRIO directors go to local sources such as other staff members, professional 
association publications, reports by state, district schools, their own college, as well as staff 
members of partner schools and districts to seek information that will help them to plan their 
services. These local sources were reported more frequently than more remote, less personally 
known sources such as academic journals (Figure 1). 


0% 20% 40% 60% 80% 100% 


Staff members 
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Report by state, district, schools 

Your college or your organization 

Staff of partner schools, districts, 
Educational newspapers, websites 
Institutional Research, Enrollment Office 
Academic Journal 
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What Works Clearinghouse 


= Yes @No M&Noresponse 


When it comes to helpfulness of information sources, TRIO directors also see staff members, 
staff of partner schools and districts and, their own institutional research offices as more helpful 
than academic journals, general newspapers and the What Works Clearinghouse (WWC) (Figure 
wae 
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The 2015 Student Support Services (SSS) competition requirements and the experience of grant 
writing seemed to have had an influence on the SSS directors’ likelihood of reporting that they 
sought information from the What Works Clearinghouse (WWC). Figure 3 presents the 
breakdown of response to a question that asked if they looked into WWC by program. Over 80 
percent of SSS directors reported they looked into the WWC, while the rate was much lower 
among other programs. 
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Use of Research and Evaluation. Asked whether they had used research or evaluation in the 
past twelve months to inform their practice, 88 percent of the TRIO directors responded yes. 
The most common topics of the study reported were: 


program activities and interventions (69 percent); 

study of their own TRIO program (46 percent); 

other TRIO program (44 percent), and 

college access programs that are not TRIO programs (43percent). 


When we asked the research methods used for the studies that the TRIO directors remembered as 
most useful, only SSS program directors cited Randomized Controlled Trials (RCTs). This may 
derive from that the competitive preference priorities of the 2015 SSS grant competition required 
applicants to propose effective services and approaches that are supported by RCTs or rigorous 
quasi-experimental studies. Figure 4 below presents the types of study designs cited by SSS 
program and non-SSS programs. 
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Using a study for preparing a grant proposal was more frequently mentioned by SSS directors 
than any other program directors. SSS directors mentioned using a study for grant preparation 
22 times in the open-ended question on how they used studies, while other programs (Talent 
Search) directors mentioned using a study for grant-writing only three times. 


Challenges of Using Research and Evaluation. The survey asked TRIO directors if they 
came across challenges for using research that are often mentioned in the literature. For TRIO 
directors, “finding relevant studies” are the most common challenge. Compared to finding 
relevant studies, factors such as finding that the studies were too complicated, or not 
understanding how to interpret the findings were less commonly reported challenges for the 
TRIO directors. Figure 5 below presents their responses. 
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Evaluation of Own Program. In TRIO programs, internal staff often take responsibility to 
evaluate their own programs. In the survey, of 80 respondents, 65 (81 percent) reported their 
program is evaluated, and 92 percent of program directors reporting evaluations reported that the 
director or program staff conducts evaluations. There were 13 directors reporting that they have 
an external evaluator, but all of them reported that either the director or staff also conducts 
evaluations. 


Figure 6 below presents types of evaluations and evaluation approaches used by the TRIO 


programs. 


The most common evaluation is summative evaluation, followed by formative 


evaluation and needs assessment. Logic models were less utilized, as only 10 programs reported 


they either create or revise a logic model for their program. 
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Summative evaluation 7 
Formative evaluation 45 
Needs assessment 44 
Logic model 10 
Includes comparison group(s) 31 
Case study 10 
Randomized controlled trial 1 


Participant satisfaction/feedback 54 


Over one-third of TRIO programs reported their evaluation includes a comparison group (see 
Figure 7). The majority of these programs are SSS programs. Programs that focus on college 
students, rather than pre-college, are more likely to include comparison groups. 
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The most common data the programs use for evaluation were from: 


e Annual performance reports (94 percent of programs that conduct evaluation reported 
using them) and 
e Participant surveys (77 percent). 


Use of Data and Evaluation. Overall, TRIO programs report high use of data and 
evaluation. About two-thirds (67 percent) of programs reported they hold a data review meeting 
at least once a month, and all programs that reported conducting evaluations of their own 
program reported they used the evaluation. The most common use of the evaluation is program 
or strategy planning and for the grant applications. About half of the programs also reported 
using evaluation to communicate to their stakeholders and participants. One-quarter (25 percent) 
of programs reported using evaluations for proposals to other funders. 


140 


Plan/revise program 
Plan/revise activity strategies 
TRIO grant application 
Report to leadership/board 
Staffing decision 
Share best practices/lessons 
Resource allocation decision 
Report to stakeholders 
Report to participants 
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Scaling up strategies 


Collaboration with Other Organizations. Since TRIO programs are not required to 
allocate a specific budget for program evaluation, utilizing existing resources or partnering with 
other organizations or units seem to be one of the ways to conduct evaluation. Thus, we asked if 
their program collaborates with other units or organizations to study or evaluate their own 
programs. Just over one-third (38 percent) of programs reported they collaborated with other 


units or organizations. SSS and McNair programs are more likely to report having collaborators 
or partners. 
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The most common collaborators are institutional research offices. Examples of study projects 
are as follows: 


e Working with the Institutional Research office to compare SSS students, SSS-eligible 
students and non-SSS-eligible students (reported by SSS program) 

e With the Institutional Research Office and with university professors, studying “what are 
viable new initiatives to expedite students from remediation?” 

e With the assessment office, conducting a study using propensity score analysis to 
compare matched sample of participants and eligible non participants’ retention 
differences. (SSS program) 

e With the Institutional Research Office, studying effective interventions and use of 
resources (Upward Bound Program) 


Evaluation Questions. Respondents were asked an open-ended question regarding what they 
would most like to know if they had unlimited resources for evaluating or studying their 
program. TRIO directors reported a variety of study questions, with most centering on how best 
to serve students generally and also how to serve particular students. These questions ranged 
from understanding the best practices and student expectations to identifying effective services 
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for particular types of students, better management of multiple programs, the true impact of the 
program, and the long-term impact of the program on students and institutions. Below are some 
examples of the study questions listed by the TRIO directors. 


e Questions about needs, expectations, and standards. 


o “What are students’ expectations?” “What should staff know to be effective? 
What professional development is needed?’’(SSS) 

o “How to evaluate my program against state and national norms for retention 
purposes” (SSS) 

o “What encourages low-income, first-gen, and underrepresented students to pursue 
graduate school and persist to the PhD?” (McNair) 


e Questions to identify effective services, strategies, and programs. 


o “What educational methods work best with middle-aged veterans preparing to 
attend college?” (VUB) 

o “What are evidence-based best practices for student success programs working 
with this population (First-generation, low-income, students with disabilities)? 
What are other effective services that can be used to engage, retain, graduate, 
and/or transfer this population?” (SSS program) 

o “To isolate variables associated with SSS intervention and correlate them with 
program participant academic performance and retention” (SSS) 

o “What are research-proven methods for early intervention for our demographic of 
students whom are currently not making adequate progress in school or state 
mandated testing? What are research proven methods to increase college 
retention rates for our demographic of students?” (Upward Bound Math and 
Science) 

o “Effective and efficient academic support approaches for college readiness” 
(EOC) 

o “What support or services are relevant for students? (e.g. group advising or 
individual advising? Students with trauma, older students, students at risk)” 
(Talent Search, SSS) 

o “What is our program’s ROI for retention or graduation rates at our institution?” 
(SSS) 

o “Cost effectiveness” (Talent Search) 

e Questions about program management 


o “How to effectively run a program that travels daily 30-60 min to reach 5 target 
schools? How to effectively work with the rising costs of university housing, 
dining room reservations? How to structure the coordinated impact of running 
two or three TRIO programs” (Upward Bound) 


e Questions to identify “true” impact 
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o “How to determine program impact in high school completion and student grades 
vs impact at the school level? Would like to be able to document the true 
academic benefit of intervention services for students” (Upward Bound) 


e Questions about the long-term impact of the program in the broader community. 


o “How does the existence of the SSS program impact perceptions of 
socioeconomic diversity at my institution?” (SSS) 

o “What are the unanticipated outcomes for TRIO program participants, i.e., lower 
incarceration rates, higher employment?” (Upward Bound) 


Effective Strategies. Another important purpose of this survey was to map a landscape of 
practitioner knowledge. To the question, “please list strategies or practices that you utilize 
regularly to make your program effective and successful,” 95 percent of respondents answered. 
The respondents provided multiple activities or strategies. Below, we present examples of 
responses in the following two categories: a) practices that are program activities, services, and 
b) practices that are associated with evaluations. 


e Strategies and practices that focus on how to implement services. 


O 


“Establish partnership with community organization who serve a similar population 
as EOC and TRIO target population” (EOC) 

“Offer advisement services, onsite, at high schools, GED programs, homeless 
shelters, and community organizations, helping participants apply for financial aid 
and college admissions at their respective locations. Provide college admissions and 
FAFSA workshop at various locations in the EOC serving communities. Follow up 
with participants either at EOC main office or out in the community” (EOC) 

“Take students to research conferences to either see other students presenting 
research or to present their own research. Work with students on writing for 
publication, practicing and studying for the GRE, preparing for interviews, etiquette 
and other workshops” (McNair program) 

“Ongoing personal learning, staff training and development, proactive review of 
formative assessment and data points throughout the semester and year, review of 
summative assessment and evaluation to shape program changes for the upcoming 
year” (SSS) 

“Learning communities involving students in high-impact practices integrating 
academic support into classes requiring participation in activities and using future 
scholarship award as leverage” (SSS) 

“Proactive intervention models for students on or at risk of academic probation or 
students returning from academic probation. Providing a foundation of community 
atmosphere so our students feel they have a safe place to come and talk to staff about 
a wide range of concerns. Utilizing proactive interventions for students who are on 
the early alert and on the financial aid purge list have been fruitful.” (SSS) 
“Consistent contact with participants during target school visits. Inclusion of parents 
in programs. Summer programming that is focused and comprehensive. Providing 
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students with an array of academic and social development activities.” (Talent 
Search) 

“Research based intervention for low-income, first generation students based on the 
work of Suzanne Bouffard and Eric Jensen among others. Incorporate those 
strategies into summer program and staff training” (Upward Bound) 

“Mandatory turning for students who have a C or below in each high school course... 
who have not passed their state mandated testing. During our summer program, we 
prescreen and place students who have not been successful on their state testing into 
rigorous remediation and then work with the schools to provide a retesting period 
after their summer remediation. We are also working on becoming a test center to 
remediate and retest all of our students from the seven different pilot schools that we 
serve” (Upward Bound Math and Science Program) 

“We consult colleagues in programs who serve the same kind of population we do. 
We base our work on our experience and try always to have at least one highly 
experienced staff employed. We listen to our participants’ expressed needs” 
(Veterans Upward Bound) 


e Use of data, evaluation, research, and feedback loop. 


O 


“1. Review of program enrollment and service utilization, 2. Regularly assess 
outcomes data. 3. Work closely with our campus IR department to analyze trends and 
maintain compliance with ED outcomes, 4. Inter-departmental collaboration, 
especially with athletics, student affairs, career and technical programs, financial aid 
and admissions” (SSS) 

“T hire tribal members when possible. I have meetings in the tribal community versus 
the school system. I always seek the tribal councils’ blessings for my initiatives or 
activities” (Talent Search) 

“Constant evaluation and re-assessment. Create opportunities where our students are 
exposed to different learning strategies. Create opportunities to talk with students to 
understand their needs. Trust is important in working with people. When our 
students know that we have their best interest at heart, they will begin to open up and 
possibly see that their lives can change. Reinforcement is the key” (Upward Bound) 
“We conduct semi-annual evaluations with our school partners, we solicit feedback 
from our students after every event, our dialogue with families informs our 
communication strategies” (Upward Bound) 

“T complete an annual program review that is instituted by the college. It requires that 
I look at my program’s data and outcomes through a different lens. I look more at the 
qualitative instead of just the quantitative data” (Upward Bound Math and Science) 


Discussion 


The purpose of the Exploratory Landscape survey was to gather information about the use of 
research and evaluation in TRIO programs and their evaluation practice. The survey also aimed 
to collect effective practices. The descriptive information from the survey was to promote future 
discussions about how to better promote research and evaluation use among TRIO programs and 
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how to promote useful evaluations. In this section, drawing upon other studies and the findings 
from this survey, I present potential future discussion points. 


Are the Survey Findings Reflective of TRIO Programs? Responses to the Exploratory 
Landscape survey project an image of TRIO programs as highly engaged with evaluation 
because programs regularly review data, program directors and staff members evaluate their 
programs, and they use findings. When our survey responses are compared with the findings 
from a national survey on nonprofit organizations, TRIO programs are close to the national 
average in the frequency of conducting evaluations. In the national study in 2012, 78 percent of 
nonprofit organizations that have annual budgets of $500,000 or less, which is comparable to 
most of TRIO programs, conduct evaluations. In our survey, TIRO programs (81 percent) 
reported conducting an evaluation. However, it is very important to note that this response may 
not be reflective of all TRIO programs. We expect that those TRIO programs most interested in 
evaluation were also those that-completed our survey. Furthermore, our survey sample is COE 
membership, which may include respondents whose institutions have a long history of TRIO 
programs or TRIO directors who are more engaged with college access and success missions. 
As we discussed, there may be a potential bias in our respondents, and there is a possibility that 
TRIO programs in our sample are more engaged in evaluation than TRIO programs in general. 
There is a need to review whether the findings in this survey reflect TRIO programs in general. 
COE will be able to disseminate survey findings and to facilitate discussions. 


How Did the SSS Proposal Requirement of Proposing Services or Practices That 
Met the WWC Change the Way TRIO Programs Use Research and Evaluation for 
their Project Planning? The survey responses indicate that the most recent SSS program 
application and competition requirements might have led SSS directors to seek information from 
WWC and to review studies that meet WWC’s evidence standards. This presents an interesting 
opportunity for TRIO community to better understand how this new requirement changed the 
way TRIO programs use research and evaluation for designing its program and services. The 
current survey does not provide this information. The open-ended responses made us to think 
that SSS programs used studies for grant writing. Since the use of RCT and quasi-experimental 
studies require implementing sites to prepare the best context to make the interventions work, it 
is important for the TRIO community to have a better understanding of how to make potentially 
effective interventions actually work. From the perspective of the field of study of evaluation 
use, it is worth investigating how this type of “imposed use” of evaluation (Weiss, Murphy- 
Graham & Birkeland, 2005, p. 12) influences how TRIO programs use research and evaluation 
in the long term. Does the exposure to WWC facilitate enlightenment or instrumental use? 


How Do TRIO Programs Design Evaluation? The survey also identified interesting 
aspects of evaluation practice among TRIO programs. 


The survey found that despite the fact that 81 percent of respondents reported conducting 
evaluations, only 15 percent reported using a logic model, which seems to be lower than the 
national survey results. A survey of non-profit organizations found 41 percent of organizations 
created and revised a logic model within the past 12 months, and 60 percent of organizations 
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have logic models (Morariu, Pankaj & Emery, 2012). This relatively low usage of logic models 
may stem from the fact that the evaluation is conducted by internal staff, as one of the reasons of 
drawing a logic model is to support evaluators understanding of the project they are evaluating. 
Another reason may be that the logic model is not required in grant proposals. There is a need to 
better understand how TRIO programs identify evaluation questions, design, and implement 
evaluation. 


What Makes College Programs (SSS and McNair) More Likely to Collaborate with 
Other Units to Conduct Study of their Programs than Others? The survey also found 
that SSS programs and McNair programs were more likely to report collaborating with other 
units to support studies of their own programs. Types of institution where programs are hosted, 
such as 2-year or 4-year colleges, do not seem to be the main factor. In this survey, of 41 
programs that reported collaborating with other units, 21 were 4-year and the rest were either 2- 
year or other organizations. The commonality of McNair and SSS programs are they are both 
college programs, but we do not know why they are more likely to collaborate with other units 
than other programs. A further study to find out why and how these programs could collaborate 
with other units will be helpful for the TRIO community. The first step is conducting interviews 
with the programs. 


What is the Best Way to Present Effective Strategies Reported by TRIO Programs so 
that the Document will be Useful for TRIO Programs, but also be Able to Present 
the Landscape of Knowledge about Effective Strategies? This exploratory survey 
collected information about many effective strategies and practices. The volume of responses 
provided by the respondents suggests that each program has developed its own unique set of 
strategies and practices. The programs reported multiple ways of knowing the effective 
strategies--for example, receiving feedback from students, conducting evaluations, and 
monitoring program performance. The next step for the ASHE-Pell Collaboration is mapping 
these diverse practices that are embedded in the context in a way that can be communicated to 
other programs. 


Conclusion 


This exploratory landscape survey is a first attempt to describe the way TRIO programs use 
research and evaluation for programming, their evaluating practices and effective strategies 
among TRIO programs. While the survey response rate was low, this survey is the first of its 
kind to systematically collect information from TRIO directors on how they use research and 
evaluation for programing, and the findings have raised several areas for further studies. As 
Competitive Preference Priorities (CPPs) have become a part of grant requirements, the Pell 
Institute continues to collect information on how TRIO programs respond to CPP requirements 
and how programs are using studies to design interventions. 


* This survey, The State of Evaluation, by Innovation Network, was conducted in 2012 on 546 representatives of 
501c3 nonprofit organizations. The final report is available online: http://stateofevaluation.org 
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BY HEATHER ROWAN-KENYON, MIKA YAMASHITA, AND MARGARET CAHALAN 


The essays included in Reflections on Connecting Research and Practice in College Access and 
Success Programs are diverse but have in common that they highlight the value of researcher 
and practitioner collaboration in working together to promote better outcomes for students. The 
essays also address the important but complex practical question of “how researchers and college 
access practitioners can best work together to inform one another and promote better outcomes 
for students.” 


This collection presents examples of multiple forms of “research” conducted in TRIO and GEAR 
UP and other college access and success programs. While the current policy focus on federal 
programs is associated with strong pressure to use evidence-based interventions that are 
supported by quantitative random assignment, “rigorous” research (as reported by Cahalan and 
Yamashita), the examples of research included in this work are not limited to this type of 
research. The essays report studies that use qualitative, mixed-methods, and participatory action 
research. Furthermore, while some “researchers” conducted more theoretical research, others 
conducted specific program evaluations (Bell, et. al, Kimball, et al). TRIO programs have a long 
history of requiring extensive performance measurement at the individual student level, often 
involving years of tracking all the students served by the program. Essays written by college 
access professionals (Mullen and Felix) demonstrate how practitioners can use available 
information to produce internal knowledge that supports program improvement in the absence of 
external researchers. Some essays (Beasley and Cahalan) present lessons learned from the 
challenges of collaboration between researchers and practitioners such as in some of the national 
evaluations of TRIO programs. 


Although researchers and practitioners are frequently viewed as two distinct groups, the essays 
show that this distinction is becoming blurred in an era of increased technological tools for 
processing and analysis of data, and the increased demands for routine use of data to inform 
programming. This distinction also becomes artificial when we define research broadly as 
“activity that employs systematic, empirical methods to address a specific question” (Cahalan). 
Performance monitoring could also be seen as research as it collects information systematically 
to answer a specific question, i.e., if a program meets pre-determined performance standards. 


The authors also showcase the wide variety in who is doing this work. Often we see that 
academic researchers have a strong ongoing connection to college access and success programs, 
either as a staff member or even as a participant during their own pre-college/college experience. 
While one might automatically default to a researcher as a professor working in academia, in 
actuality researchers doing this important work are part of a much larger group, including 
program evaluators, staff of intermediary organizations, and university administrators. 


In reflecting on this work, we recognize that the dichotomous designation of “researchers” or 
“practitioners” is an oversimplification. For one thing, it leaves out the important group of 
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policy decision makers that influences the context and rules under which the college access and 
success programs operate, and that provides research funding and at times mandates the exact 
types of research to be conducted. As Figure 1 below illustrates, the identities of “practitioners” 
and “researchers” involve considerable actors and overlap in practice. 


Policy Decision Makers 
in democracy: 
government at federal, 
state local, (legislative, 
executive, judicial), 
advocacy groups, 
stakeholder 
representatives, non- 
goverment furcders) 


Professional 

Researchers 

employed to 

research social 
science and 
education 
topics(academics, 
contractors, non- 
profits, advocacy, 
professional groups) 


As described by Perna’s essay, however, researchers and practitioners continue to be different 
just as research tasks and the day-to-day services provided by practitioners continue to be 
different. Researchers and practitioners work under different norms and incentives. They are 
expected to produce different types of knowledge. The essays written by researchers describe 
challenges that researchers face when they negotiate research questions with program staff 
(Venegas, Kimball, et al.). The researchers need to be able to design a study that can respond to 
a program’s interest in order to keep their support for the research, and to improve the quality of 
the study. At the same time, their study design needs to respond to their particular research 
interest. Program staff needs to be able to ask questions that are meaningful for their programs 
(Mullen, Yamashita). One of the important questions we need to ask, beyond researchers 
acquiring buy- in from program staff to conduct research, is how both researchers and 
practitioners can formulate research questions that are meaningful for both groups. 


Below we list some recommendations that have emerged from the diverse essays that we hope 
will improve future collaboration between researchers and practitioners. 
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1. 


Think About Adequate Research Needs During the Program Planning and 
Funding Process But Decouple Competitive Funding Decisions From Results of 
Research Evaluations of Particular Interventions. One of the key recommendations 
is that the research question should be meaningful to programs and there should be adequate 
resources to adequately address the research questions; however, it is important to ensure that 
funding for research does not reduce or supplant ongoing program services. Ethically, when 
college access and success services are at stake, it is important to not deny services to any 
student simply for the sake of a study. To ensure an honest and open study, it is also 
important to decouple funding for future needed program services from the outcomes of 
evaluations of a particular intervention. TRIO practitioners and researchers should 
collaborate on creating feasible, ethical, valid, and accurate research and evaluation plans, as 
highlighted by Laura Perna and Bell and colleagues. These collaborations can empower 
TRIO practitioners to have a voice in the research process and ensure that the research will 
meet relevant knowledge needs for TRIO practitioners. For researchers, the voice of 
practitioners is essential to ensure the quality and validity of the data (as discussed by Bell) 
and to determine whether the research plan is feasible and ethical, as detailed by Cahalan in 
her essay on lessons learned from the Mathematica Upward Bound Study. Yamashita added 
that TRIO practitioners need to share their own logics of evidence use that reflect how TRIO 
professionals work in practice, since randomized control trials provide little information 
about actual interventions for individual program improvement. More research is needed to 
reflect how TRIO programs actually work. 


Seek Authentic Collaborations from the Time of Conceptualization. One 
challenge that many researchers and practitioners face is that collaborations can often be one- 
sided. A researcher may show up on the doorstep of a program director asking to study 
program students in a certain way; or a program staff member may reach out to a researcher 
with a request to collect data in a particular way. While these transactional relationships may 
work, they are not true authentic collaborations built from a point of mutual understanding. 
As Kiyama discusses, collaboration develops through relational networks of people who 
share the same social justice concerns. Vergas discusses researchers need to ask meaningful 
questions by first listening to program staff. 


Build on Existing Relationships and Particular Contexts. Structure Frequent 
Communication. While the collaboration between researchers and TRIO practitioners is 
an asset, the process is rarely easy and it takes effort to build and maintain a successful 
partnership. As Kiyama and Kimball and colleagues shared, partnerships that develop out of 
previously formed relationships can be most helpful. As Beasley shared, these relationships 
can work to develop interventions that are “custom fit” to the particular program and/or 
campus context. As was highlighted in all of the essays focused on collaboration, open and 
frequent communications and dialogues are needed between the staff and researchers 
throughout the life of the partnership. Kristan Venegas and others also highlighted that the 
key to developing authentic collaborations is to build trust between the participants and the 
researchers. One way to do this is through some shared areas of responsibility such as data 
collection. Both the researchers and practitioners will need to be flexible, as shared by Bell 
and colleagues. These partnerships and shared responsibilities that may be necessary to 
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make data collection “less burdensome and more efficient” can also help break down silos 
and increase the likelihood of much higher quality and more comprehensive data collection. 


Include both Traditional and Innovative Research Methods in Preparation of 
Both College Access Professionals and Academic Researchers. The essays 
highlighted the need to help in developing the competency of traditional and innovative 
research and evaluation methods in the academic training of TRIO professionals, as well as 
the need for researchers’ own training to include innovative research methods such as 
collaborative, participatory, culturally responsive, and action research. Beasley called for 
development of a course of study to prepare TRIO professionals to administer effective 
college access and success programs. He highlighted the new certificate program, College 
Access and Success Programs Graduate Certificate, a joint initiative between Colorado State 
University and the Council for Opportunity in Education, as one example to provide current 
college access professionals with academic courses and meaningful qualifications directly 
related to their work. Researchers can also be better prepared for entering these 
collaborations through the inclusion of a variety of new methodologies such as collaborative, 
participatory and action research, which values and promotes authentic collaboration 
between researcher, practitioner and participant. 


Developing Reflective Awareness of the Importance of Differences in 
Positionality in Influencing Outlook. For both practitioners and researchers, it is 
important to be self-aware and reflect on the importance of their different positions in 
influencing their interpretations of the data and its implications. This can be an asset in 
deepening and strengthening the research and practice. 


Disseminate Research in an Accessible Way. As seen in our attempt to map the 
landscape of practitioner knowledge related to research and evaluation, there is a disconnect 
between the published research in this area and what practitioners are actually reading. 
Research results must be made available and accessible to TRIO practitioners and 
policymakers. In our landscape analysis presented by Yamashita, there are signs that that in 
the recent SSS funding cycle, practitioners accessed studies from the What Works 
Clearinghouse as required in their grant writing. However, it is not clear what practitioners 
took away from the research findings to design their programs. Researchers need to think 
about disseminating scholarship in ways that are available and accessible, beyond peer- 
reviewed scholarly publications, which is usually the most valuable product for these 
researchers, and use multiple outlets in the form of trade publications, reports, executive 
summaries, and other alternative formats to get the findings out there to increase the 
possibility of recommendations for practice actually getting implemented. Felix reports that 
a professional community of practice network is a promising path to inform practitioners 
about evidence-based practices. 


. Acknowledge Program Improvement and Excellence is the Goal. As can be seen in 
all of the essays, practitioners and researchers alike strive to produce programs and conduct 
research with the intent of improving practice to benefit students. Bell and colleagues 
emphasize the importance of formative evaluation as findings could contribute to improve 
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programs and services throughout the process. Mullin presented how leadership and 
effective management can facilitate using data to make mid-course improvements and to 
explain performance and highlight program improvement. As Cahalan highlights in her 
piece about evaluation errors in random assignment studies, one cannot assume that 
implementation of “rigorous” study will be free from errors and that these issues need to be 
acknowledged early. Checks and balances need to be in place in examining the data, and if 
mistakes are found, they need to be corrected. 


Recognize the Dynamic Quality of both Practice and Research. As the essays 
illustrate, just as there is not always a clear distinction between researchers and practitioners, 
there is also no clear linear pathway from research to practice—or from practice to research. 
Rather, one might think of multiple pathways with continuous feedback loops. Practice is 
dynamic, whether it is informed by research or not--- with a necessity for continuous 
adaptation and responding to new circumstances. Useful research also needs to build in a 
dynamic and adaptive quality. A study begun in one year may find that the program has 
changed substantially by the next year for reasons out of the control of the researchers. 


Consider the Importance of the Federal Context and Competitive Grant Making 
Process, and Reconsider the Current Federal Approach to Evidence Use. The 
nature of operating as a federally funded entity is an important aspect to be considered when 
we discuss how to improve collaboration between researchers and TRIO practitioners. 
Because TRIO is a federally funded program, the role of federal government needs to be 
examined. As Beasley and Cahalan described, TRIO experiences a strong top-down policy 
pressure to use a specific type of research evidence and to participate in specific research 
studies that may affect individual program funding and the existence of the programs 
themselves. Despite this strong pressure to be evaluated, the focus of data collection at each 
program is on performance outcome monitoring rather than evaluation or a focus on a 
particular set of interventions that might be studied. Because of this structure, federal-level 
decisions and guidance on what constitutes evidence and how programs should use evidence 
has a direct consequence for each program. Government’s evaluation policy matters for 
TRIO programs. Practitioners put a great deal of time into their competitive proposals, 
conforming to government requirements, but while they may meet the requirements on paper, 
these responses are often not feasible in practice. The CPP requirement in all of the 2015- 
2017 competitions was based on the idea that “interventions” supported by “rigorous 
evidence” if implemented in the TRIO programs would improve the program outcomes for 
students. As Cahalan described in her two essays, this is a very limited view of how 
evidence can or should inform practice and what constitutes practice in a particular context. 
As shown in the landscape survey and in the essays in this volume, there are multiple ways 
evidence feeds into various aspects of practice. For example, Bell and colleagues’ essay 
reports on the use of research to improve program processes, and Kimball and colleagues’ 
essay reports how the program saw Action Research as helpful. Essays written by TRIO 
practitioners and researchers highlight the complex policy context in which TRIO programs 
operate and how federal government’s evidence initiatives, evaluation resource allocation, 
and monitoring requirement present challenges to programs concerned with conducting 
useful research that might meet each individual program’s information needs. The essays 
presented here suggest there is a need to reconsider the current federal government’s 
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approach to research use and to providing guidance to TRIO professionals in how best to 
use research. 


10. Focus on Potentially Useful Questions that are Feasible to Address with the 
Resources and Information Available to be Collected or Compiled. Cahalan 
illustrated how questions being asked by the government are often not easily answered in a 
valid manner and how this can lead to serious threats to validity of the study and can 
potentially be harmful to program stakeholders. Asking the right questions, ones that are 
aligned with the feasible data in the end is one of the most important collaborative actions 
that can occur between researchers and practitioners. 


In conclusion, this collection of essays is intended to advance conversations and actions 
surrounding effective ways to link research and practice on the role of college access and success 
programs in increasing college access and attainment. We hope that the essays promote thinking 
and dialogue about how we can all work collaboratively to create meaningful improvements in 
higher education access and success. 
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